This library is released every month, usually at the first week of month.

Changelog in 2021. This is changelog for share module since v0.22.0 until v0.32.0.

Changelog in 2020. This is changelog for share module since v0.12.0 until v0.21.0.

Changelog from 2018 to 2019. This is changelog for share module since v0.1.0 until v0.11.0.

share v0.42.0 (2022-12-12)

New features

lib/http: add methods PutForm and PutFormData on Client

The PutForm method send the PUT request with params set in body using content type "application/x-www-form-urlencoded".

The PutFormData method send the PUT request with params set in body using content type "multipart/form-data".

Chores

lib/websocket: fix test for the Go 1.20

In the next Go release (1.20), parsing URL with invalid percent escape will not throw error anymore [1].

lib/ascii: seed math/rand to fix the example for Random function

In the next Go release, the math/rand is globally seed randomly [1]. This cause our example for Random is always fail.

The fix is to seed it manually using predefined number.

share v0.41.3 (2022-11-05)

Bug fixes

lib/http: sanitize ServerUrl and request path

This is to fix double slash in case the ServerUrl end with it and the request path begin with it. For example, "http://127.0.0.1/" and "/" should send request to "http://127.0.0.1/" not "http://127.0.0.1//".

lib/ssh: try both ssh agent and IdentityFile

If ~/.ssh/config contains a Host section with IdentityFile directive specified and SSH_AUTH_SOCK environment variable is also set, but ssh_config section does not specify "IdentityAgent none" explicitly, the NewClientFromConfig returns an error saying

NewClient: ssh: handshake failed: ssh: unable to authenticate, attempted
methods [none publickey], no supported methods remain.

This changes fix this by dialing remote server twice.

ssh/sftp: set FileAttrs.name to filename

Current implementation exposes dirEntry.filename as fs.DirEntry.Name(). However fs.DirEntry.Info().Name() is always empty string.

Enhancements

lib/ssh: add private key to agent once client connected successfully

In NewClientFromConfig, if Client connect using IdentityFile instead of agent and its success, we add the private key to agent directly.

Unfortunately, since we did not know which key is being negotiated, we add all private keys parsed from IdentityFile.

ssh/config: change the method GenerateSigners to Signers

This is to make the method compatible with ssh.PublicKeysCallback.

Each parsed and unsigned IdentityFile is stored in field PrivateKeys, replacing the Signers field (which is conflict with method names).

share v0.41.2 (2022-10-10)

Bug fixes

lib/websocket: check for EAGAIN and EINTR when reading raw socket

This fix tests that sometimes fail when running with -count=X, where X > 1,

$ go test -race -count=30 -timeout=30s ./lib/websocket

Upon inspecting, when client sending larger payload, for example 65536 bytes, server sometimes only read half of them and return an error "resource temporarily unavailable" or "interrupted system call".

lib/websocket: fix possible data race on Client

The Client have method send that check if the underlying connection (conn) has been closed or not. Since the conn can be closed anytime, for example server send to the control CLOSE frame:

recv -> handleFrame -> handleClose -> Quit

we need to guard the conn with Mutex before calling send to prevent data race.

Enhancements

lib/websocket: cleanup the channel gracefulClose during Close

When calling Close, we initialize the internal channel gracefulClose so the client can check the close response from server on another goroutine serve() and signal back to Close to continue the closing process.

This channel is never closed and nil-ed again after Close which may cause resource leaks.

While at it, use sendClose to minimize duplicate code.

Chores

lib/websocket: replace handleInvalidData and BadRequest with sendClose

Both of those functions actually call send control CLOSE frame and not exported. So, instead of duplicating it, we replace it with sendClose.

lib/websocket: rewords some comment and package documentation

share v0.41.1 (2022-10-07)

Enhancements

lib/http: simplify the default user-agent that send by Client

Remove the comment string, no need to be explicit about it.

lib/http: support embedded field on UnmarshalForm
lib/test: use text/diff to compare strings on Assert

If both exp and got types are string and its longer than 50 chars, it will use the text/diff.Text to show the difference between them. The diff output is as follow,

!!! string not matched:
--++
<LINE_NUM> - "<LINE_EXP>"
<LINE_NUM> + "<LINE_GOT>"
^<COL_NUM> - "<DELETED_STRING>"
^<COL_NUM> + "<INSERTED_STRING>"

The "<LINE_NUM> - " print the line number in exp followed by line itself. The "<LINE_NUM> + " print the line number in got followed by line itself. The "^<COL_NUM> - " show the character number in exp line followed by deleted string (or string that not exist in got). The "^<COL_NUM> + " show the character number in got line followed by inserted string (or string that not exist in exp).

lib/reflect: remove prefix from returned error on DoEqual

Prefixing an error cause may cause confusion when used on lib/test.Assert. The returned error from test.Assert will print "DoEqual: …​", where user never call DoEqual in their test.

lib/test: change the Assert parameter to Writer

Since we only need to call Log and Fatalf during Assert, no need to pass the whole instance of testing.T to Assert. By changing it to Writer, we also can test the Assert.

This remove the AssertBench, because it have the same function parameters and body as Assert.

Chores

all: fix some tests with -count=X, where X>1

The fixed test are in package lib/dns, lib/http, lib/smtp, lib/git, and email/dkim.

lib/ini: add example for marshaling slice inside map[string]T

While at it, clean up some codes to make it more readable and debug-able.

lib/paseto: reformat the documentation
lib/websocket: fix possible race during testing Client

During testing the Client we use the un-exported method send, while the test cases itself may close the connection and we did not guard it.

text/diff: rewrite the test again by reading from files

The goal is to remove dependency to lib/test so we can use text/diff in the lib/test in the future.

share v0.41.0 (2022-09-04)

Breaking changes

lib/json: remove solidus (slash) from being escaped/un-escaped

The standard json package does not escape the solidus, even though the RFC said so. Someone also report this as an error in RFC 3159 by removing solidus from list of escaped characters but the author itself reject it.

Bug fixes

lib/memfs: fix SIGSEGV when node is deleted when being watched

The panic is caused by the item in slice of Childs is being removed during iteration.

To fix this, we remove the childs on the second iteration after we remove any sub directories inside them.

New features

lib/text: add custom MarshalJSON to type Chunk and Line

Chores

lib/memfs: simplify checking for symlink

Instead of calling filepath.EvalSymlink and Lstat, call os.Stat directly to the symlink system path.

This also fix the modTime not currently set to the original file when creating Node from symlink-ed file.

lib/http: increase time sleep waiting for server on example endpoint

On container, sometimes the test fail with the following error

2022-08-28 19:32:21 UTC DefaultErrorHandler: POST /error/custom:
    Custom error
2022/08/28 19:32:22 Do: Get "http://127.0.0.1:7016/?":
    dial tcp 127.0.0.1:7016: connect: connection refused
FAIL	github.com/shuLhan/share/lib/http	1.583s

This was caused by server is not ready yet to accept connection when testing executed.

text/diff: rewrite the test using test.Data

Using test.Data provide much more readable input and outputs and simplify modifying the test data instead of manually define the expected output in struct.

_doc: cleaning up the documentation

In the _doc, we remove generated HTML files.

In the index, we add link to README and section about Development that include links to repository, issues, and patches.

In the README, we reformat it to use AsciiDoc markup, remove the sanitizer library, add CLI for epoch, ini, and xtrk.

lib/totp: cleaning up the codes

This changes replace ":=" with explicit variable declarations and use raw string literal whenever possible.

share v0.40.0 (2022-08-05)

Breaking changes

lib/memfs: set the Root SysPath to the first MemFS instance on Merge

Previously, calling Merge(…​), set the merged MemFS Root.SysPath to "..". Since we allow the TryDirect to access the file directly (if its set to true), this may cause the file system leaks if returned MemFS set this flag to true.

To prevent that, we set the SysPath to the first MemFS SysPath.

lib/memfs: rename Option field Development to TryDirect

This changes the usage of Development flag.

TryDirect define a flag to bypass file in memory. If its true, any call to Get will try direct read to file system.

This flag has several use cases. First, to test serving file system directly from disk during development. Second, to combine embedded MemFS instance with non-embedded instance. One is reading content from memory, one is reading content from disk directly.

New features

_bin: add script to run Go benchmark

The go-bench.sh accept two arguments: the method or function to benchmark, default to "."; and benchmark number, default to current timestamp YYYYmmDD-HHMM.

_bin: add script to run Go test and generate HTML coverage

The script accept one single argument: the path to package to be tested. If its empty default to current directory and sub-directories.

_bin: add script go-mod-tip.sh

The go-mod-tip shell script get and print the latest Go module version based on the last tag and the latest commit hash from the current working directory.

This command usually used to fix go.mod due to force commit.

cmd/epoch: print the weekday in local and UTC time
cmd/epoch: add flag to parse time from RFC3339 and RFC1123 format

The flag for RFC1123 comes with two options one with string timezone (-rfc1123) and one with numeric time zone (-rfc1123z).

cmd/ini: a CLI to get and set values in the INI file format

This is the CLI that implements the lib/ini for getting and setting the key’s value from INI file.

lib/test: implement Data, a type to load formatted file for helping test

Data contains predefined input and output values that is loaded from file to be used during test.

The data provides zero or more flags, an optional description, zero or more input, and zero or more output.

The data file name must end with ".txt".

The data content use the following format,

[FLAG_KEY ":" FLAG_VALUE LF]
[LF DESCRIPTION]
LF
">>>" [INPUT_NAME] LF
INPUT_CONTENT
LF
"<<<" [OUTPUT_NAME] LF
OUTPUT_CONTENT

The data can contains zero or more flag. A flag is key and value separated by ":". The flag key must not contain spaces.

The data may contain description.

The line that start with "\n>>>" defined the beginning of input. An input can have a name, if its empty it will be set to "default". An input can be defined multiple times, with different names.

The line that start with "\n<<<" defined the beginning of output. An output can have a name, if its empty it will be set to "default". An output also can be defined multiple times, with different names.

Bug fixes

lib/ini: fix parsing and saving multi line variables

Previously, if INI file contains multi line variables, for example

key = a \
b

The Get and saved value is "a \tb", where it should be "a b" for Get and "a \\\n\t\b" again when saved.

This changes require refactoring how the variable’s value is parsed and stored. A variable value is parsed and stored from character after "=" until new line or comment as raw value, and the real value is derived by trimming white spaces, handle escaped character and double quotes.

lib/ini: fix marshaling pointer to nil field

If the field is pointer, the code will thrown panic if its point to nil struct or print "<invalid reflct.Value>" for String.

lib/memfs: ignore error on Get when calling node Update

If node exist in memory, error on Update does not means the file is not exist. The node may have been embedded and then merged with other MemFS instance with Development flag set to true.

Enhancements

lib/dns: add field SOA to the ServerOptions

The SOA field defined the root authority for all zones and records served under the Server.

lib/http: add server options to generate index.html automatically

If the EnableIndexHtml in the ServeOptions enabled, server generate list of files inside the requested path as HTML.

lib/ini: support escaped double-quote and colon in tag subsection

A colon : is escaped using double backslash \\, for example a:b\\:c:d contains section a, subsection b:c, and variable d.

A double quote " is escaped using triple backslash, for example (\\\").

lib/ini: handle marshaling slice of time.Time

-

share v0.39.0 (2022-07-03)

Breaking changes

all: move lib/sanitize.HTML to net/html.Sanitize

Since the sanitize package only contains HTML function, and the html package already exist, we move the function into html package.

New features

lib/mlog: add method Close to MultiLogger

The Close method flush and close all log forwarders. Any write to a closed MultiLogger will be ignored.

This changes require adding sync.Mutex to mark if the instance has been closed or not; which affect createMultiLogger and defaultMLog to return a pointer to prevent copy on Mutex.

lib/clise: implement json.Marshaler on Clise

The MarshalJSON method convert the Clise into slice by calling Slice and then convert it into JSON.

lib/reflect: add function Marshal

The Marshal function marshal the obj value to []byte by calling one of the method: MarshalBinary, MarshalJSON, or MarshalText; in respective order.

If obj implement one of the method with valid signature, it will return (out, nil, true); unless there is an error.

If the method signature invalid it will return (nil, err, false).

If obj is nil or none of the method exist it will return (nil, nil, false).

net/html: add function NormalizeForID

Given an input string, The NormalizeForID normalize it to HTML ID. The normalization follow Mozilla specification [1] rules,

  • it must not contain whitespace (spaces, tabs etc.),

  • only ASCII letters, digits, '_', and '-' should be used, and

  • it should start with a letter.

The NormalizeForID do this normalization,

  • An empty string is equal to "_".

  • Any other unknown characters will be replaced with '_'.

  • If the input does not start with letter, it will be prefixed with '_', unless it start with '_'.

  • All letters converted to lower case.

lib/http: add function to unmarshal url.Values using tag form:

UnmarshalForm read struct fields tagged with form: from out as key and set its using the value from url.Values based on that key. If the field does not have form: tag but it is exported, then it will use the field name, in case insensitive.

Only the following types are supported: bool, int/intX, uint/uintX, floatX, string, []byte, or type that implement BinaryUnmarshaler (UnmarshalBinary), json.Unmarshaler (UnmarshalJSON), or TextUnmarshaler (UnmarshalText).

A bool type can be set to true using the following string value: "true", "yes", or "1".

If the input contains multiple values but the field type is not slice, the field will be set using the first value.

It will return an error if the out variable is not set-able (the type is not a pointer to a struct). It will not return an error if one of the input value is not match with field type.

lib/reflect: implement Set function to set reflect.Value by string

The Set function set the obj value by converting the string val from parameter to the obj type.

If the obj is an interface or struct, its value will be set by calling Unmarshal.

It will return an error if,

  • obj is not setable, variable is passed without pointer or pointer not initialized.

  • val is overflow

  • obj Kind is Invalid, Array, Chan, Func, Map, or UnsafePointer.

lib/reflect: add function Unmarshal

The Unmarshal function set the obj value by calling one of the method: UnmarshalBinary, UnmarshalJSON, or UnmarshalText; in respective order.

Just like reflect, the obj value must be pointer to initialized variable (&T) or pointer-to-pointer to uninitialized variable (**T).

If obj implement one of the method, it will return (true, nil) if there is no error.

If none of the method exist on obj, it will return (false, nil).

lib/reflect: add function Tag to simplify lookup on struct’s field tag

Given a StructField and the name of tag, return the tag’s value and options inside the tag. The options is any string after tag’s value, separated by comma. For example, given the following field definition

F `tag:"name,opt1, opt2"`

It will return (name, [opt1 opt2], true).

If the field is exported but does not have tag, it will return the field name (as is without converting to lower case) in val with hasTag is false: (Name, nil, false).

If the field is un-exported it will return empty val with hasTag is false ("", nil, false).

Enhancements

lib/memfs: update the template format

Replace ":=" with "var" and realign the field assignments.

net/html: use inline replacement to clean up white spaces

Instead of using bytes.Replace, three times, iterate the plain text manually to clean up the white and multiple spaces.

Benchmark result,

name        old time/op    new time/op    delta
Sanitize-8    4.27µs ±10%    2.64µs ±13%  -38.21%  (p=0.000 n=10+10)

name        old alloc/op   new alloc/op   delta
Sanitize-8    4.84kB ± 0%    4.45kB ± 0%   -7.94%  (p=0.000 n=10+10)

name        old allocs/op  new allocs/op  delta
Sanitize-8      13.0 ± 0%       6.0 ± 0%  -53.85%  (p=0.000 n=10+10)
lib/mlog: minimize allocation when generating log

Instead of using two bytes.Buffer pool, use one; and add space after time and prefix by writing to buffer directly instead of allocating new arguments to Fprintf.

Benchmark result,

name           old time/op    new time/op    delta
MultiLogger-8    3.97µs ± 3%    3.68µs ± 2%   -7.43%  (p=0.008 n=5+5)

name           old alloc/op   new alloc/op   delta
MultiLogger-8      510B ± 1%      300B ± 1%  -41.13%  (p=0.008 n=5+5)

name           old allocs/op  new allocs/op  delta
MultiLogger-8      10.4 ± 6%       3.4 ±18%  -67.31%  (p=0.008 n=5+5)
lib/dns: use Shutdown to stop DoH server

Using Shutdown allow active connection not interrupted but it may cause delay when restarting the server.

While at it, set the doh and dot server instance to nil to release the resource, in case the Server need to start again.

lib/websocket: realign all struct to minimize allocations

Changes,

  • Client: from 176 to 144 (-32 bytes)

  • ClientManager: from 64 to 40 (-24 bytes)

  • Frame: from 72 to 56 bytes (-16 bytes).

  • Handshak: from 160 to 120 bytes (-40 bytes).

  • Request: from 88 to 72 (-16 bytes)

  • Response: from 40 to 24 (-16 bytes)

  • route: from 48 to 32 (-16 bytes)

  • Server: from 72 to 64 (-8 bytes)

  • ServerOptions: from 104 to 96 (s-8 bytes)

Plus other structs in the tests.

share v0.38.0 (2022-06-05)

This release update the minimum Go version to 1.17.

Breaking changes

  • lib/dns: move all caches operations from Server to Caches type

    Previously all caches operation are tied to the Server type.

    In order to separate the responsibilities between server and caches, we move all caches operations to Cache type.

  • lib/dns: change the Zone SOA field type from ResourceRecord to RDataSOA

    Using the RDataSOA type directly minimize interface check and conversion.

Enhancements

  • lib/dns: replace Ticker with Timer on Caches’ worker

    Since the worker call time.Now() inside the body, we can minimize it by using Timer.

  • lib/dns: export the Caches type and field on Server

    The idea is move all server’s caches operations (methods) to this type later.

  • lib/dns: split storage between internal and external caches

    Previously, the indexed caches for internal (records from hosts or zone files) and external (records from parent name servers) are stored inside single map.

    This changes split those internal and external caches into two maps, so any operation on one caches does not affect the other one, and vice versa.

  • lib/dns: return the removed record on caches RemoveCachesByRR

    If the record being removed found on caches, it will return it; otherwise it will return nil without error.

  • lib/dns: disable JSON marshaling Zone Records field

    On service that manage many zones, providing an API to fetch list of zones only will return large payload if we include the Records field in the response.

    So, it is recommended to provide another API to fetch the records on specific zone.

  • lib/dns: print the field Value on ResourceRecord Stringer instead of rdlen

  • lib/dns: export the zoneRecords type

    Since the Zone type is exported and its contains exported field Records with type zoneRecords, then that field type should also exported.

  • lib/dns: return the deleted record on HostsFile RemoveRecord

    Previously, the RemoveRecord method on HostsFile return a boolean true if the record found.

    This changes the return type to the ResourceRecord being deleted, to allow the caller inspect and/or print the record.

Chores

  • all: rewrite all codes to use "var" instead of ":="

    Using ":=" simplify the code but we lose the type. For example,

    v := F()

    The only way we know what the type of v is by inspecting the function F. Another disadvantages of using ":=" may cause extra variables allocation where two or more variables with same type is declared inside body of function where it could be only one.

    While at it, we split the struct for test case into separate type.

  • lib/memfs: format comment in embedded Go template according to gofmt tip

    In the next gofmt (Go v1.19), the comment format does not allow empty lines "//" at the top and bottom of the comment.

    This changes make the generated Go code from Embed method to match as close as possible with output of gofmt.

share v0.37.0 (2022-05-09)

Breaking changes

  • lib/dns: refactor Server RemoveCachesByNames to return removed Answer

    Previously, RemoveCachesByNames does not return anything, its only print the domain name being removed if debugging level is set to >= 1.

    This changes rewrite the RemoveCachesByNames to return list of Answer being removed to allow the caller to inspect and/or print the Answer.

  • lib/memfs: change the Watch method to accept struct

    Previously, we assume that the list of files being Watch-ed is same with the list of files in Includes. This may not be correct. For example, we may want to watch "*.ts" files only but did not want it to be included during GoEmbed.

    This changes introduce list of pattern for files to be watched in the WatchOptions.Watches field. If this field is empty, only files match the Includes filter will be watched.

New features

  • lib/dns: add method to get the record in HostsFile by name and/or value

    The Get method return the first resource record that match with domain name and/or value. The value parameter is optional, if its empty, then only the first record that match with domain name that will be returned.

    If no record matched, it will return nil.

    While at it, reformat the hosts_file comments with next Go 1.19 format.

  • lib/dns: add method CachesClear to remove all caches

  • lib/net: add method to populate query on ResolvConf

    Given a domain name to be resolved, the PopulateQuery generate list of names to be queried based on registered Domain and Search in the resolv.conf file.

    The domain name itself will be on top of the list if its contains any dot.

  • lib/dns: add function to create new client using name server URL

    The NewClient create new DNS client using the name server URL. The name server URL is defined in the same format as ServerOptions’s NameServer.

    The function also accept second parameter: isInsecure, which is only usable for DNS over TLS and DNS over HTTPS.

Bug fixes

  • lib/ini: fix panic when marshaling unexported field with type struct

    While at it, split the example for marshaling and unmarshaling struct into separate examples.

  • lib/memfs: make the Node’s addChild to be idempotent

    If the same Node’s Path already exists on the Childs, adding another Node with same Path should not add the Node to the Childs.

Enhancements

  • lib/ini: implement marshaling and unmarshaling map with struct element

    For a field F with type map[K]S ini:"sec", where K is string and S is a struct or pointer to struct element, marshaling the field F will result in the following ini format,

    [sec "K"]
    <S.Field.Tag> = <S.Field.Value>

    Each field in struct S unmarshaled normally as "key = value".

    This rule is also applied when unmarshalling from ini text into map[K]V.

    This implementation allow multiple section with dynamic subsections as key.

Chores

  • all: reformat all codes using gofmt 1.19 (the Go tip)

  • all: replace any usage of ioutil package with os or io

    Since Go 1.16, the ioutil package has been deprecated. This changes replace any usage that use functions from ioutil package with their replacement from package os or package io.

share v0.36.0 (2022-04-03)

Breaking changes

  • lib/memfs: update the file mode and/or content on DirWatcher

    Previously, the DirWatcher only forward the NodeState if the file being watched is deleted or modified.

    This changes the DirWatcher handle it internally. If the file is deleted it will be removed from internal MemFS instance. If the file is updated it will update the mode or content of that file in the MemFS.

  • lib/memfs: changes the DirWatcher and Watcher to use channel

    Previously, we use a callback model to propagated changes. This model has its advantages and disadvantages.

    The advantages is there is no limit of queue when the changes need to be propagated to the caller. The disadvantages of that the watcher needs to wait for callback to finish before continue processing. One can run it under goroutine, but it may cause race if the caller does not handle update properly and it does not guarantee the goroutine process it in FIFO. We can see this on the unit test of NewWatcher, we needs to use sync.WaitGroup to properly check one changes before processing the order.

    This commit changes the DirWatcher and Watcher to use channel, like the one in time.Ticker.

  • all: move the DirWatcher and Watcher types from io to memfs

    There are two reasons why we move them. First, DirWatcher and Watcher code internally depends on the memfs package, especially on Node type. Second, we want to add new Watch method to MemFS which depends on package io. If we do that, there will be circular imports.

New features

  • lib/http: implement handler to check each request to Server Memfs

    The FSHandler define the function to inspect each GET request to Server MemFS instance. The node parameter contains the requested file inside the memfs.

    If the handler return true, server will continue processing the node (writing the Node content type, body, and so on).

    If the handler return false, server stop processing the node and return immediately, which means the function should have already handle writing the header, status code, and/or body.

  • lib/memfs: add method to stop the Watch

    The StopWatch method stop watching for update, from calling Watch.

  • lib/xmlrpc: add method to get boolean field value on Value

    The GetFieldAsBoolean return the struct’s field value by its key as bool type.

  • lib/memfs: add method Watch to MemFS

    The Watch method create and start a DirWatcher that ready to be consumed.

    This is to simplify watching an existing MemFS instance because the internal fs inside the DirWatcher is not exported.

Enhancements

  • lib/http: use package mlog for logging

    In case the consumer of lib/http package use mlog for logging, the log will be written to their predefined writers.

    In case they did not use mlog, the log will written to stdout and stderr.

Bug fixes

  • lib/memfs: check for possible nil on Get

    In case the instance of memfs is set to nil (for example, the root directory being watched is deleted on DirWatcher), the Get method will cause panic after the next update on content of root directory.

  • lib/xmlrpc: use %v to convert non-string type on GetFieldAsString

    Previously, if GetFieldAsString is called and the struct field type is not string, it will return "%s(<type>=<value>)" instead of the value in string.

    This commit fix this issue by using %v to convert non-string type.

Chores

  • lib/memfs: differentiate prefix on MemFS’s Update and Node’s Update

  • email/dkim: remove amazonses.com domain from test cases

    The domain now return invalid public key record, so we removed them to make the test passed for now.

  • lib/memfs: move the test for NewWatcher and DirWatcher as example

    With this we do one thing (write testing) and output two things (testing the code and give an example for code).

share v0.35.0 (2022-03-04)

Breaking changes

  • lib/email: change the Header and Body fields on Message to non-pointer.

    The idea is to minimize GC pressure on system with many messages, minimize checking for nil value, and make an empty Message ready to use without any initialization.

  • lib/smtp: refactoring NewClient to use struct instead of parameters.

    Previously, to create new client one must pass three parameters to NewClient function: localName, remoteURL, and insecure. If we want to add another parameters in the future, it will cause the function signature changes.

    This changes simplify creating NewClient by passing single struct with new parameters: AuthUser, AuthPass, and AuthMechanism. If both AuthUser and AuthPass is not empty, the NewClient will authenticate the connection, minimize number of step on the caller.

  • lib/smtp: rename Mechanism to SaslMechanism.

New features

  • cmd/sendemail: command line interface to send an email.

    The sendemail command is proof of concept on how to use lib/email and lib/smtp to write and send email through SMTP.

  • cmd/xtrk: command line interface to uncompress and/or un-archive file.

    xtrk accept single file to uncompress and/or archived into a directory output dir". If directory output "dir" is not defined, it will be set to current directory.

    The compression and archive format is detected automatically based on the following file input extension:

    • .bz2: decompress using bzip2.

    • .gz: decompress using gzip.

    • .tar: unarchive using tar.

    • .zip: unarchive using zip.

    • .tar.bz2: decompress using bzip2 and unarchive using tar.

    • .tar.gz: decompresss using gzip and unarchive using tar.

    The input file will be removed on success.

Enhancements

  • lib/dns: increase the default UDP packet size to 1232.

    The value is based on recommendation by https://dnsflagday.net/2020/ to prevent IP fragmentation when supporting EDNS message.

  • lib/memfs: export the Remount method.

    The Remount method reset the memfs instance to force rescanning the files again from file system.

  • lib/email: set the Date and Message-ID on Message Pack.

    Calling Pack now set the Date header if its not exist, using the local time; and the message-id header if its not exist using the following format:

    <epoch>.<random-8-chars>@<local-hostname>

    The random-8-chars is Seed-ed from Epoch(), so does the boundary.

  • lib/email: make Message Pack works with single text or HTML part.

    Previously, the Pack method generate multipart/alternative message only.

    Since the Message now can set the body text and HTML, without using NewMultipart, the Pack method need to be able to accommodate this.

  • lib/email: add methods to modify Message.

    Previously, a Message can be created only using NewMultipart, which generate message with text and HTML.

    This changes add methods to compose a Message: AddCC, AddTo, SetBodyHtml, SetBodyText, SetCC, SetFrom, SetSubject, and SetTo.

  • lib/email: set the header Date field on NewMultipart.

    The Date field value is set to current time on the system that generated the message.

    The date format is set to "Mon, 2 Jan 2006 15:04:05 -0700" according to RFC 5322 section 3.3.

  • lib/smtp: add status codes from RFC 4954.

    The following status codes are added,

    • 432: StatusPasswordTransitionNeeded, from section 4.7.12.

    • 454: StatusTemporaryAuthFailure, from section 4.7.0.

    • 534: StatusAuthMechanismTooWeak, from section 5.7.9.

  • lib/mlog: make the Outf method always add new line at the end.

    One of common mistakes when using logging library is to put the new line "n" at the end of format string, which cause delayed output written to Stdout (the OS wait for "n" as signal for printing).

    This changes check new line to every call of Outf method and add it if its not exist.

    If the caller need to call Outf multiple times before ending it with new line, they should handle it manually by storing into temporary buffer first and call Outf at the end.

  • lib/memfs: add option CommentHeader to EmbedOptions.

    The CommentHeader option allow user to define custom header to the Go generated file. The string value is not checked, whether it’s a comment or not, it will rendered as is.

  • lib/ini: make the Marshal on map field sorted by keys

    Given the following struct,

    type ADT struct {
    	Amap map[string]string `ini:"section:sub"`
    }

    and ini text,

    [test "map"]
    c = 3
    b = 2
    a = 1

    Unmarshal-ing the text into ADT and then Marshal-ing it again will result in unpredictable keys order.

    This changes fix this issue by sorting the keys on ADT.Amap on Marshal-ing, to make the written output predictable.

Bug fixes

  • lib/io: fix DirWatcher not removing old files on rename.

    Previously, if a sub-directory being watched by DirWatcher is renamed, the old directory does not get removed from field dirs.

    This commit fix this issue by deleting the sub directory on unmpSubdirs.

    While at it, guard any read/write to dirs field with mutex to prevent data race.

  • lib/dns: check for possible index out of range when unpacking RR.

    There is a possibility that record data (rdata) length inside the packet is greater than length of packet itself. Some of the reasons are corrupted packet from server or packet poisoning (attacking the DNS server by sending invalid packet).

    This changes fix this issue by checking the index and rdata length with the length of packet before consuming the rdata itself.

Chores

  • lib/smtp: provide an example of how to create MailTx from email package.

    If one read the current documentation on how to use the Client.SendTx, there is a missing link on how to create and populate MailTx.

    This changes provide the example using the email package to generate the MailTx Data.

  • lib/mlog: change default mlog instance to non-pointer.

    Since the default mlog instance is a global variable, using non-pointer give advantages on minimize GC pressure.

share v0.34.0 (2022-02-05)

Breaking changes

  • lib/sql: make the table migration customizable

    In the method Migrate() we add parameter "tableMigration" which define the name of table where the state of migration will be saved.

    If its empty default to "_migration".

New features

  • lib/os: implement function to Extract compressed and/or archived file

    The Extract function uncompress and/or unarchive file from fileInput into directory defined by dirOutput. This is the high level API that combine standard archive/zip, archive/tar, compress/bzip2, and/or compress/gzip.

    The compression and archive format is detected automatically based on the following fileInput extension:

    • .bz2: decompress using compress/bzip2.

    • .gz: decompress using compress/gzip.

    • .tar: unarchive using archive/tar.

    • .zip: unarchive using archive/zip.

    • .tar.bz2: decompress using compress/bzip2 and unarchive using archive/tar.

    • .tar.gz: decompress using compress/gzip and unarchive using archive/tar.

    The output directory, dirOutput, where the decompressed and/or unarchived file stored. will be created if not exist. If its empty, it will set to current directory.

    On success, the compressed and/or archived file will be removed from the file system.

  • lib/http: implement method Download() on Client

    The Download method get a resource from remote server and write it into DownloadRequest.Output (a io.Writer).

Enhancements

  • lib/websocket: return error if parameter is empty on RegisterTextHandler

    Previously, the RegisterTextHandler method return nil if method, target, or handler parameter is not set. This may cause confusion and hard to debug handler when no connection receive but the RegisterTextHandler does not have any error.

Chores

  • lib/http: change the test port for testing HTTP server

    Previously, the test port for HTTP server is set to 8080 and may conflict with any service that running on the local (due to common use of 8080).

    This changes it to 14832 and we make the full server address stored as global variable so any tests can references it.

share v0.33.0 (2022-01-09)

Happy New Year!

Three years has passed since the first release of this multi-libraries (or Go module), and we have released at least 33 new features with several bugs here and there.

For anyone who use this module, I hope it help you, as the module name intended "share", and sorry if its too many breaking changes.

Live long and prosper! See you again next year.

New features

  • cmd/gofilemode: new command to decode the Go file mode

    The Go has their own file mode that works across all operating system. The file mode is represented by uint64, the command line will convert it to fs.FileMode and print each possible flag on it including the permission.

  • lib/sql: make the TruncateTable run with cascade and restart identity

    On table that contains foreign key, truncate without cascade may cause the method fail.

    Also, since TruncateTable is, and should be only, used on testing, any identity columns, for example serial, should be reset back to its initial value. On PostgreSQL this means the truncate table is with "RESTART IDENTITY".

  • cmd/epoch: command line to print and parse Unix timestamp

    Program epoch print the current time (Unix seconds, milliseconds, nanoseconds, local time, and UTC time) or the time based on the epoch on first parameter. Usage,

    epoch <unix-seconds|unix-milliseconds|unix-nanoseconds>

    Without a parameter, it will print the current time. With single parameter, it will print the time based on that epoch.

Breaking changes

  • lib/http: refactoring NewClient to accept single struct

    Previously, the NewClient function accept three parameters: serverURL, http.Header, and insecure. If we want to add another parameter, for example timeout it will cause changes on the function signature.

    To prevent this changes in the future, we change it now. The NewClient now accept single struct.

    While at it, we add option to set Timeout.

    The Timeout affect the http Transport Timeout and TLSHandshakeTimeout. The field is optional, if not set it will set to 10 seconds.

  • lib/http: remove field memfs.Options in ServerOptions

    This options is duplicate with Memfs.Opts.

  • lib/websocket: add "ok" return value on ClientManager Context

    The ok return value will be true if the context exist or false otherwise.

  • lib/memfs: remove field ContentEncoding from EmbedOptions and Node

    The original idea for option ContentEncoding in EmbedOptions and Node is to save spaces, compressing the content on disk on embedding and doing transport, when the MemFS instance is used to serve the (embedded) contents of file system.

    This option turns out break the HTTP content negotiation [1] of accept-encoding header, if the HTTP server does not handle it properly, which default Go HTTP server does not.

    In order to prevent this issue in the future, for anyone who use the memfs for serving static HTTP contents, we remove the options and store the embedded content as is and let the HTTP server handle how the compression by itself.

  • lib/email: refacforing ParseMailbox

    This commit changes the signature of ParseMailbox by returning no error.

Bug fixes

  • lib/memfs: skip mount if the Root node has been initialized

  • lib/websocket: fix race conditition on handleText

    Instead of accessing the ctx field directly, call the Context() method to prevent data race.

  • lib/sql: check for EOF on loadSQL

    There is probably a regression in Go that cause ioutil.ReadAll return io.EOF, while it should not, because the documentation said that

    A successful call returns err == nil, not err == EOF.

    But in this, using http.FileSystem, the ioutil.ReadAll now return EOF and we need to check it to make the migration can run without an error.

Enhancements

  • lib/io: realign all structs

    The struct realign, save the occupied of struct size in the memory,

    • DirWatcher: from 184 to 144 bytes (-40 bytes)

    • Reader: from 16 to 8 bytes (-8 bytes)

    • Watcher: from 32 to 24 bytes (-8 bytes)

  • lib/http: realign all structs

    Changes, * Client: from 56 to 48 bytes (-8 bytes) * CORSOptions: from 104 to 88 bytes (-16 bytes) * Endpoint: from 64 to 32 bytes (-32 bytes) * EndpointRequest: from 72 to 56 bytes (-16 bytes) * route: from 56 to 32 bytes (-24 bytes)

    Other changes is struct on unit tests.

  • lib/memfs: add method Init

    The Init provided to initialize MemFS instance if its Options is set directly, not through New() function.

  • lib/memfs: embed the Embed options and GenFuncName

    This is to make the instance of memfs initialize from init is reusable.

  • lib/memfs: realign struct Node, Options, PathNode, and on unit tests

    The realign save storage spaces on struct,

    • Node: from 240 to 224 bytes (-16 bytes)

    • Options: from 112 to 104 bytes (-8 bytes)

    • PathNode: from 16 to 8 bytes (-8 bytes)

  • lib/email: realign the struct Mailbox

    This changes the storage size from 80 to 72 bytes (-8 bytes).

Chores

  • github/workflows: remove step to get dependencies

    The Go module should handle the dependencies automatically.

  • github/workflows: set go version to 1.17.6

  • lib/email: convert the unit test for ParseMailbox to examples

    Since the ParseMailbox is public we can provide an examples and test at the same times.