This library is released every month, usually at the first week of month.
Changelog in 2021. This is changelog for share module since v0.22.0 until v0.32.0.
Changelog in 2020. This is changelog for share module since v0.12.0 until v0.21.0.
Changelog from 2018 to 2019. This is changelog for share module since v0.1.0 until v0.11.0.
share v0.42.0 (2022-12-12)
New features
- lib/http: add methods PutForm and PutFormData on Client
-
The PutForm method send the PUT request with params set in body using content type "application/x-www-form-urlencoded".
The PutFormData method send the PUT request with params set in body using content type "multipart/form-data".
Chores
- lib/websocket: fix test for the Go 1.20
-
In the next Go release (1.20), parsing URL with invalid percent escape will not throw error anymore [1].
- lib/ascii: seed math/rand to fix the example for Random function
-
In the next Go release, the math/rand is globally seed randomly [1]. This cause our example for Random is always fail.
The fix is to seed it manually using predefined number.
share v0.41.3 (2022-11-05)
Bug fixes
- lib/http: sanitize ServerUrl and request path
-
This is to fix double slash in case the ServerUrl end with it and the request path begin with it. For example, "http://127.0.0.1/" and "/" should send request to "http://127.0.0.1/" not "http://127.0.0.1//".
- lib/ssh: try both ssh agent and IdentityFile
-
If ~/.ssh/config contains a Host section with IdentityFile directive specified and SSH_AUTH_SOCK environment variable is also set, but ssh_config section does not specify "IdentityAgent none" explicitly, the NewClientFromConfig returns an error saying
NewClient: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain.
This changes fix this by dialing remote server twice.
- ssh/sftp: set FileAttrs.name to filename
-
Current implementation exposes dirEntry.filename as fs.DirEntry.Name(). However fs.DirEntry.Info().Name() is always empty string.
Enhancements
- lib/ssh: add private key to agent once client connected successfully
-
In NewClientFromConfig, if Client connect using IdentityFile instead of agent and its success, we add the private key to agent directly.
Unfortunately, since we did not know which key is being negotiated, we add all private keys parsed from IdentityFile.
- ssh/config: change the method GenerateSigners to Signers
-
This is to make the method compatible with ssh.PublicKeysCallback.
Each parsed and unsigned IdentityFile is stored in field PrivateKeys, replacing the Signers field (which is conflict with method names).
share v0.41.2 (2022-10-10)
Bug fixes
- lib/websocket: check for EAGAIN and EINTR when reading raw socket
-
This fix tests that sometimes fail when running with -count=X, where X > 1,
$ go test -race -count=30 -timeout=30s ./lib/websocket
Upon inspecting, when client sending larger payload, for example 65536 bytes, server sometimes only read half of them and return an error "resource temporarily unavailable" or "interrupted system call".
- lib/websocket: fix possible data race on Client
-
The Client have method send that check if the underlying connection (conn) has been closed or not. Since the conn can be closed anytime, for example server send to the control CLOSE frame:
recv -> handleFrame -> handleClose -> Quit
we need to guard the conn with Mutex before calling send to prevent data race.
Enhancements
- lib/websocket: cleanup the channel gracefulClose during Close
-
When calling Close, we initialize the internal channel gracefulClose so the client can check the close response from server on another goroutine serve() and signal back to Close to continue the closing process.
This channel is never closed and nil-ed again after Close which may cause resource leaks.
While at it, use sendClose to minimize duplicate code.
Chores
- lib/websocket: replace handleInvalidData and BadRequest with sendClose
-
Both of those functions actually call send control CLOSE frame and not exported. So, instead of duplicating it, we replace it with sendClose.
- lib/websocket: rewords some comment and package documentation
share v0.41.1 (2022-10-07)
Enhancements
- lib/http: simplify the default user-agent that send by Client
-
Remove the comment string, no need to be explicit about it.
- lib/http: support embedded field on UnmarshalForm
- lib/test: use text/diff to compare strings on Assert
-
If both exp and got types are string and its longer than 50 chars, it will use the text/diff.Text to show the difference between them. The diff output is as follow,
!!! string not matched: --++ <LINE_NUM> - "<LINE_EXP>" <LINE_NUM> + "<LINE_GOT>" ^<COL_NUM> - "<DELETED_STRING>" ^<COL_NUM> + "<INSERTED_STRING>"
The "<LINE_NUM> - " print the line number in exp followed by line itself. The "<LINE_NUM> + " print the line number in got followed by line itself. The "^<COL_NUM> - " show the character number in exp line followed by deleted string (or string that not exist in got). The "^<COL_NUM> + " show the character number in got line followed by inserted string (or string that not exist in exp).
- lib/reflect: remove prefix from returned error on DoEqual
-
Prefixing an error cause may cause confusion when used on lib/test.Assert. The returned error from test.Assert will print "DoEqual: …", where user never call DoEqual in their test.
- lib/test: change the Assert parameter to Writer
-
Since we only need to call Log and Fatalf during Assert, no need to pass the whole instance of testing.T to Assert. By changing it to Writer, we also can test the Assert.
This remove the AssertBench, because it have the same function parameters and body as Assert.
Chores
- all: fix some tests with -count=X, where X>1
-
The fixed test are in package lib/dns, lib/http, lib/smtp, lib/git, and email/dkim.
- lib/ini: add example for marshaling slice inside map[string]T
-
While at it, clean up some codes to make it more readable and debug-able.
- lib/paseto: reformat the documentation
- lib/websocket: fix possible race during testing Client
-
During testing the Client we use the un-exported method send, while the test cases itself may close the connection and we did not guard it.
- text/diff: rewrite the test again by reading from files
-
The goal is to remove dependency to lib/test so we can use text/diff in the lib/test in the future.
share v0.41.0 (2022-09-04)
Breaking changes
- lib/json: remove solidus (slash) from being escaped/un-escaped
-
The standard json package does not escape the solidus, even though the RFC said so. Someone also report this as an error in RFC 3159 by removing solidus from list of escaped characters but the author itself reject it.
Bug fixes
- lib/memfs: fix SIGSEGV when node is deleted when being watched
-
The panic is caused by the item in slice of Childs is being removed during iteration.
To fix this, we remove the childs on the second iteration after we remove any sub directories inside them.
New features
- lib/text: add custom MarshalJSON to type Chunk and Line
Chores
- lib/memfs: simplify checking for symlink
-
Instead of calling filepath.EvalSymlink and Lstat, call os.Stat directly to the symlink system path.
This also fix the modTime not currently set to the original file when creating Node from symlink-ed file.
- lib/http: increase time sleep waiting for server on example endpoint
-
On container, sometimes the test fail with the following error
2022-08-28 19:32:21 UTC DefaultErrorHandler: POST /error/custom: Custom error 2022/08/28 19:32:22 Do: Get "http://127.0.0.1:7016/?": dial tcp 127.0.0.1:7016: connect: connection refused FAIL github.com/shuLhan/share/lib/http 1.583s
This was caused by server is not ready yet to accept connection when testing executed.
- text/diff: rewrite the test using test.Data
-
Using test.Data provide much more readable input and outputs and simplify modifying the test data instead of manually define the expected output in struct.
- _doc: cleaning up the documentation
-
In the _doc, we remove generated HTML files.
In the index, we add link to README and section about Development that include links to repository, issues, and patches.
In the README, we reformat it to use AsciiDoc markup, remove the sanitizer library, add CLI for epoch, ini, and xtrk.
- lib/totp: cleaning up the codes
-
This changes replace ":=" with explicit variable declarations and use raw string literal whenever possible.
share v0.40.0 (2022-08-05)
Breaking changes
- lib/memfs: set the Root SysPath to the first MemFS instance on Merge
-
Previously, calling Merge(…), set the merged MemFS Root.SysPath to "..". Since we allow the TryDirect to access the file directly (if its set to true), this may cause the file system leaks if returned MemFS set this flag to true.
To prevent that, we set the SysPath to the first MemFS SysPath.
- lib/memfs: rename Option field Development to TryDirect
-
This changes the usage of Development flag.
TryDirect define a flag to bypass file in memory. If its true, any call to Get will try direct read to file system.
This flag has several use cases. First, to test serving file system directly from disk during development. Second, to combine embedded MemFS instance with non-embedded instance. One is reading content from memory, one is reading content from disk directly.
New features
- _bin: add script to run Go benchmark
-
The go-bench.sh accept two arguments: the method or function to benchmark, default to "."; and benchmark number, default to current timestamp YYYYmmDD-HHMM.
- _bin: add script to run Go test and generate HTML coverage
-
The script accept one single argument: the path to package to be tested. If its empty default to current directory and sub-directories.
- _bin: add script go-mod-tip.sh
-
The go-mod-tip shell script get and print the latest Go module version based on the last tag and the latest commit hash from the current working directory.
This command usually used to fix go.mod due to force commit.
- cmd/epoch: print the weekday in local and UTC time
- cmd/epoch: add flag to parse time from RFC3339 and RFC1123 format
-
The flag for RFC1123 comes with two options one with string timezone (-rfc1123) and one with numeric time zone (-rfc1123z).
- cmd/ini: a CLI to get and set values in the INI file format
-
This is the CLI that implements the lib/ini for getting and setting the key’s value from INI file.
- lib/test: implement Data, a type to load formatted file for helping test
-
Data contains predefined input and output values that is loaded from file to be used during test.
The data provides zero or more flags, an optional description, zero or more input, and zero or more output.
The data file name must end with ".txt".
The data content use the following format,
[FLAG_KEY ":" FLAG_VALUE LF] [LF DESCRIPTION] LF ">>>" [INPUT_NAME] LF INPUT_CONTENT LF "<<<" [OUTPUT_NAME] LF OUTPUT_CONTENT
The data can contains zero or more flag. A flag is key and value separated by ":". The flag key must not contain spaces.
The data may contain description.
The line that start with "\n>>>" defined the beginning of input. An input can have a name, if its empty it will be set to "default". An input can be defined multiple times, with different names.
The line that start with "\n<<<" defined the beginning of output. An output can have a name, if its empty it will be set to "default". An output also can be defined multiple times, with different names.
Bug fixes
- lib/ini: fix parsing and saving multi line variables
-
Previously, if INI file contains multi line variables, for example
key = a \ b
The Get and saved value is "a \tb", where it should be "a b" for Get and "a \\\n\t\b" again when saved.
This changes require refactoring how the variable’s value is parsed and stored. A variable value is parsed and stored from character after "=" until new line or comment as raw value, and the real value is derived by trimming white spaces, handle escaped character and double quotes.
- lib/ini: fix marshaling pointer to nil field
-
If the field is pointer, the code will thrown panic if its point to nil struct or print "<invalid reflct.Value>" for String.
- lib/memfs: ignore error on Get when calling node Update
-
If node exist in memory, error on Update does not means the file is not exist. The node may have been embedded and then merged with other MemFS instance with Development flag set to true.
Enhancements
- lib/dns: add field SOA to the ServerOptions
-
The SOA field defined the root authority for all zones and records served under the Server.
- lib/http: add server options to generate index.html automatically
-
If the EnableIndexHtml in the ServeOptions enabled, server generate list of files inside the requested path as HTML.
- lib/ini: support escaped double-quote and colon in tag subsection
-
A colon
:
is escaped using double backslash\\
, for examplea:b\\:c:d
contains sectiona
, subsectionb:c
, and variabled
.A double quote
"
is escaped using triple backslash, for example (\\\"
). - lib/ini: handle marshaling slice of time.Time
-
-
share v0.39.0 (2022-07-03)
Breaking changes
- all: move lib/sanitize.HTML to net/html.Sanitize
-
Since the sanitize package only contains HTML function, and the html package already exist, we move the function into html package.
New features
- lib/mlog: add method Close to MultiLogger
-
The Close method flush and close all log forwarders. Any write to a closed MultiLogger will be ignored.
This changes require adding sync.Mutex to mark if the instance has been closed or not; which affect createMultiLogger and defaultMLog to return a pointer to prevent copy on Mutex.
- lib/clise: implement json.Marshaler on Clise
-
The MarshalJSON method convert the Clise into slice by calling Slice and then convert it into JSON.
- lib/reflect: add function Marshal
-
The Marshal function marshal the obj value to []byte by calling one of the method: MarshalBinary, MarshalJSON, or MarshalText; in respective order.
If obj implement one of the method with valid signature, it will return (out, nil, true); unless there is an error.
If the method signature invalid it will return (nil, err, false).
If obj is nil or none of the method exist it will return (nil, nil, false).
- net/html: add function NormalizeForID
-
Given an input string, The NormalizeForID normalize it to HTML ID. The normalization follow Mozilla specification [1] rules,
-
it must not contain whitespace (spaces, tabs etc.),
-
only ASCII letters, digits, '_', and '-' should be used, and
-
it should start with a letter.
The NormalizeForID do this normalization,
-
An empty string is equal to "_".
-
Any other unknown characters will be replaced with '_'.
-
If the input does not start with letter, it will be prefixed with '_', unless it start with '_'.
-
All letters converted to lower case.
-
- lib/http: add function to unmarshal url.Values using tag
form:
-
UnmarshalForm read struct fields tagged with
form:
from out as key and set its using the value from url.Values based on that key. If the field does not haveform:
tag but it is exported, then it will use the field name, in case insensitive.Only the following types are supported: bool, int/intX, uint/uintX, floatX, string, []byte, or type that implement BinaryUnmarshaler (UnmarshalBinary), json.Unmarshaler (UnmarshalJSON), or TextUnmarshaler (UnmarshalText).
A bool type can be set to true using the following string value: "true", "yes", or "1".
If the input contains multiple values but the field type is not slice, the field will be set using the first value.
It will return an error if the out variable is not set-able (the type is not a pointer to a struct). It will not return an error if one of the input value is not match with field type.
- lib/reflect: implement Set function to set reflect.Value by string
-
The Set function set the obj value by converting the string val from parameter to the obj type.
If the obj is an interface or struct, its value will be set by calling Unmarshal.
It will return an error if,
-
obj is not setable, variable is passed without pointer or pointer not initialized.
-
val is overflow
-
obj Kind is Invalid, Array, Chan, Func, Map, or UnsafePointer.
-
- lib/reflect: add function Unmarshal
-
The Unmarshal function set the obj value by calling one of the method: UnmarshalBinary, UnmarshalJSON, or UnmarshalText; in respective order.
Just like reflect, the obj value must be pointer to initialized variable (&T) or pointer-to-pointer to uninitialized variable (**T).
If obj implement one of the method, it will return (true, nil) if there is no error.
If none of the method exist on obj, it will return (false, nil).
- lib/reflect: add function Tag to simplify lookup on struct’s field tag
-
Given a StructField and the name of tag, return the tag’s value and options inside the tag. The options is any string after tag’s value, separated by comma. For example, given the following field definition
F `tag:"name,opt1, opt2"`
It will return (name, [opt1 opt2], true).
If the field is exported but does not have tag, it will return the field name (as is without converting to lower case) in val with hasTag is false: (Name, nil, false).
If the field is un-exported it will return empty val with hasTag is false ("", nil, false).
Enhancements
- lib/memfs: update the template format
-
Replace ":=" with "var" and realign the field assignments.
- net/html: use inline replacement to clean up white spaces
-
Instead of using bytes.Replace, three times, iterate the plain text manually to clean up the white and multiple spaces.
Benchmark result,
name old time/op new time/op delta Sanitize-8 4.27µs ±10% 2.64µs ±13% -38.21% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Sanitize-8 4.84kB ± 0% 4.45kB ± 0% -7.94% (p=0.000 n=10+10) name old allocs/op new allocs/op delta Sanitize-8 13.0 ± 0% 6.0 ± 0% -53.85% (p=0.000 n=10+10)
- lib/mlog: minimize allocation when generating log
-
Instead of using two bytes.Buffer pool, use one; and add space after time and prefix by writing to buffer directly instead of allocating new arguments to Fprintf.
Benchmark result,
name old time/op new time/op delta MultiLogger-8 3.97µs ± 3% 3.68µs ± 2% -7.43% (p=0.008 n=5+5) name old alloc/op new alloc/op delta MultiLogger-8 510B ± 1% 300B ± 1% -41.13% (p=0.008 n=5+5) name old allocs/op new allocs/op delta MultiLogger-8 10.4 ± 6% 3.4 ±18% -67.31% (p=0.008 n=5+5)
- lib/dns: use Shutdown to stop DoH server
-
Using Shutdown allow active connection not interrupted but it may cause delay when restarting the server.
While at it, set the doh and dot server instance to nil to release the resource, in case the Server need to start again.
- lib/websocket: realign all struct to minimize allocations
-
Changes,
-
Client: from 176 to 144 (-32 bytes)
-
ClientManager: from 64 to 40 (-24 bytes)
-
Frame: from 72 to 56 bytes (-16 bytes).
-
Handshak: from 160 to 120 bytes (-40 bytes).
-
Request: from 88 to 72 (-16 bytes)
-
Response: from 40 to 24 (-16 bytes)
-
route: from 48 to 32 (-16 bytes)
-
Server: from 72 to 64 (-8 bytes)
-
ServerOptions: from 104 to 96 (s-8 bytes)
Plus other structs in the tests.
-
share v0.38.0 (2022-06-05)
This release update the minimum Go version to 1.17.
Breaking changes
-
lib/dns: move all caches operations from Server to Caches type
Previously all caches operation are tied to the Server type.
In order to separate the responsibilities between server and caches, we move all caches operations to Cache type.
-
lib/dns: change the Zone SOA field type from ResourceRecord to RDataSOA
Using the RDataSOA type directly minimize interface check and conversion.
Enhancements
-
lib/dns: replace Ticker with Timer on Caches’ worker
Since the worker call time.Now() inside the body, we can minimize it by using Timer.
-
lib/dns: export the Caches type and field on Server
The idea is move all server’s caches operations (methods) to this type later.
-
lib/dns: split storage between internal and external caches
Previously, the indexed caches for internal (records from hosts or zone files) and external (records from parent name servers) are stored inside single map.
This changes split those internal and external caches into two maps, so any operation on one caches does not affect the other one, and vice versa.
-
lib/dns: return the removed record on caches RemoveCachesByRR
If the record being removed found on caches, it will return it; otherwise it will return nil without error.
-
lib/dns: disable JSON marshaling Zone Records field
On service that manage many zones, providing an API to fetch list of zones only will return large payload if we include the Records field in the response.
So, it is recommended to provide another API to fetch the records on specific zone.
-
lib/dns: print the field Value on ResourceRecord Stringer instead of rdlen
-
lib/dns: export the zoneRecords type
Since the Zone type is exported and its contains exported field Records with type zoneRecords, then that field type should also exported.
-
lib/dns: return the deleted record on HostsFile RemoveRecord
Previously, the RemoveRecord method on HostsFile return a boolean true if the record found.
This changes the return type to the ResourceRecord being deleted, to allow the caller inspect and/or print the record.
Chores
-
all: rewrite all codes to use "var" instead of ":="
Using ":=" simplify the code but we lose the type. For example,
v := F()
The only way we know what the type of v is by inspecting the function F. Another disadvantages of using ":=" may cause extra variables allocation where two or more variables with same type is declared inside body of function where it could be only one.
While at it, we split the struct for test case into separate type.
-
lib/memfs: format comment in embedded Go template according to gofmt tip
In the next gofmt (Go v1.19), the comment format does not allow empty lines "//" at the top and bottom of the comment.
This changes make the generated Go code from Embed method to match as close as possible with output of gofmt.
share v0.37.0 (2022-05-09)
Breaking changes
-
lib/dns: refactor Server RemoveCachesByNames to return removed Answer
Previously, RemoveCachesByNames does not return anything, its only print the domain name being removed if debugging level is set to >= 1.
This changes rewrite the RemoveCachesByNames to return list of Answer being removed to allow the caller to inspect and/or print the Answer.
-
lib/memfs: change the Watch method to accept struct
Previously, we assume that the list of files being Watch-ed is same with the list of files in Includes. This may not be correct. For example, we may want to watch "*.ts" files only but did not want it to be included during GoEmbed.
This changes introduce list of pattern for files to be watched in the
WatchOptions.Watches
field. If this field is empty, only files match the Includes filter will be watched.
New features
-
lib/dns: add method to get the record in HostsFile by name and/or value
The Get method return the first resource record that match with domain name and/or value. The value parameter is optional, if its empty, then only the first record that match with domain name that will be returned.
If no record matched, it will return nil.
While at it, reformat the hosts_file comments with next Go 1.19 format.
-
lib/dns: add method CachesClear to remove all caches
-
lib/net: add method to populate query on ResolvConf
Given a domain name to be resolved, the PopulateQuery generate list of names to be queried based on registered Domain and Search in the resolv.conf file.
The domain name itself will be on top of the list if its contains any dot.
-
lib/dns: add function to create new client using name server URL
The NewClient create new DNS client using the name server URL. The name server URL is defined in the same format as ServerOptions’s NameServer.
The function also accept second parameter: isInsecure, which is only usable for DNS over TLS and DNS over HTTPS.
Bug fixes
-
lib/ini: fix panic when marshaling unexported field with type struct
While at it, split the example for marshaling and unmarshaling struct into separate examples.
-
lib/memfs: make the Node’s addChild to be idempotent
If the same Node’s Path already exists on the Childs, adding another Node with same Path should not add the Node to the Childs.
Enhancements
-
lib/ini: implement marshaling and unmarshaling map with struct element
For a field F with type map[K]S
ini:"sec"
, where K is string and S is a struct or pointer to struct element, marshaling the field F will result in the following ini format,[sec "K"] <S.Field.Tag> = <S.Field.Value>
Each field in struct S unmarshaled normally as "key = value".
This rule is also applied when unmarshalling from ini text into map[K]V.
This implementation allow multiple section with dynamic subsections as key.
Chores
-
all: reformat all codes using gofmt 1.19 (the Go tip)
-
all: replace any usage of ioutil package with
os
orio
Since Go 1.16, the ioutil package has been deprecated. This changes replace any usage that use functions from ioutil package with their replacement from package os or package io.
share v0.36.0 (2022-04-03)
Breaking changes
-
lib/memfs: update the file mode and/or content on DirWatcher
Previously, the DirWatcher only forward the NodeState if the file being watched is deleted or modified.
This changes the DirWatcher handle it internally. If the file is deleted it will be removed from internal MemFS instance. If the file is updated it will update the mode or content of that file in the MemFS.
-
lib/memfs: changes the DirWatcher and Watcher to use channel
Previously, we use a callback model to propagated changes. This model has its advantages and disadvantages.
The advantages is there is no limit of queue when the changes need to be propagated to the caller. The disadvantages of that the watcher needs to wait for callback to finish before continue processing. One can run it under goroutine, but it may cause race if the caller does not handle update properly and it does not guarantee the goroutine process it in FIFO. We can see this on the unit test of NewWatcher, we needs to use sync.WaitGroup to properly check one changes before processing the order.
This commit changes the DirWatcher and Watcher to use channel, like the one in time.Ticker.
-
all: move the DirWatcher and Watcher types from io to memfs
There are two reasons why we move them. First, DirWatcher and Watcher code internally depends on the memfs package, especially on Node type. Second, we want to add new Watch method to MemFS which depends on package io. If we do that, there will be circular imports.
New features
-
lib/http: implement handler to check each request to Server Memfs
The FSHandler define the function to inspect each GET request to Server MemFS instance. The node parameter contains the requested file inside the memfs.
If the handler return true, server will continue processing the node (writing the Node content type, body, and so on).
If the handler return false, server stop processing the node and return immediately, which means the function should have already handle writing the header, status code, and/or body.
-
lib/memfs: add method to stop the Watch
The StopWatch method stop watching for update, from calling Watch.
-
lib/xmlrpc: add method to get boolean field value on Value
The GetFieldAsBoolean return the struct’s field value by its key as bool type.
-
lib/memfs: add method Watch to MemFS
The Watch method create and start a DirWatcher that ready to be consumed.
This is to simplify watching an existing MemFS instance because the internal fs inside the DirWatcher is not exported.
Enhancements
-
lib/http: use package mlog for logging
In case the consumer of lib/http package use mlog for logging, the log will be written to their predefined writers.
In case they did not use mlog, the log will written to stdout and stderr.
Bug fixes
-
lib/memfs: check for possible nil on Get
In case the instance of memfs is set to nil (for example, the root directory being watched is deleted on DirWatcher), the Get method will cause panic after the next update on content of root directory.
-
lib/xmlrpc: use %v to convert non-string type on GetFieldAsString
Previously, if GetFieldAsString is called and the struct field type is not string, it will return "%s(<type>=<value>)" instead of the value in string.
This commit fix this issue by using %v to convert non-string type.
Chores
-
lib/memfs: differentiate prefix on MemFS’s Update and Node’s Update
-
email/dkim: remove amazonses.com domain from test cases
The domain now return invalid public key record, so we removed them to make the test passed for now.
-
lib/memfs: move the test for NewWatcher and DirWatcher as example
With this we do one thing (write testing) and output two things (testing the code and give an example for code).
share v0.35.0 (2022-03-04)
Breaking changes
-
lib/email: change the Header and Body fields on Message to non-pointer.
The idea is to minimize GC pressure on system with many messages, minimize checking for nil value, and make an empty Message ready to use without any initialization.
-
lib/smtp: refactoring NewClient to use struct instead of parameters.
Previously, to create new client one must pass three parameters to NewClient function: localName, remoteURL, and insecure. If we want to add another parameters in the future, it will cause the function signature changes.
This changes simplify creating NewClient by passing single struct with new parameters: AuthUser, AuthPass, and AuthMechanism. If both AuthUser and AuthPass is not empty, the NewClient will authenticate the connection, minimize number of step on the caller.
-
lib/smtp: rename Mechanism to SaslMechanism.
New features
-
cmd/sendemail: command line interface to send an email.
The sendemail command is proof of concept on how to use lib/email and lib/smtp to write and send email through SMTP.
-
cmd/xtrk: command line interface to uncompress and/or un-archive file.
xtrk accept single file to uncompress and/or archived into a directory output dir". If directory output "dir" is not defined, it will be set to current directory.
The compression and archive format is detected automatically based on the following file input extension:
-
.bz2: decompress using bzip2.
-
.gz: decompress using gzip.
-
.tar: unarchive using tar.
-
.zip: unarchive using zip.
-
.tar.bz2: decompress using bzip2 and unarchive using tar.
-
.tar.gz: decompresss using gzip and unarchive using tar.
The input file will be removed on success.
-
Enhancements
-
lib/dns: increase the default UDP packet size to 1232.
The value is based on recommendation by https://dnsflagday.net/2020/ to prevent IP fragmentation when supporting EDNS message.
-
lib/memfs: export the Remount method.
The Remount method reset the memfs instance to force rescanning the files again from file system.
-
lib/email: set the Date and Message-ID on Message Pack.
Calling Pack now set the Date header if its not exist, using the local time; and the message-id header if its not exist using the following format:
<epoch>.<random-8-chars>@<local-hostname>
The random-8-chars is Seed-ed from Epoch(), so does the boundary.
-
lib/email: make Message Pack works with single text or HTML part.
Previously, the Pack method generate multipart/alternative message only.
Since the Message now can set the body text and HTML, without using NewMultipart, the Pack method need to be able to accommodate this.
-
lib/email: add methods to modify Message.
Previously, a Message can be created only using NewMultipart, which generate message with text and HTML.
This changes add methods to compose a Message: AddCC, AddTo, SetBodyHtml, SetBodyText, SetCC, SetFrom, SetSubject, and SetTo.
-
lib/email: set the header Date field on NewMultipart.
The Date field value is set to current time on the system that generated the message.
The date format is set to "Mon, 2 Jan 2006 15:04:05 -0700" according to RFC 5322 section 3.3.
-
lib/smtp: add status codes from RFC 4954.
The following status codes are added,
-
432: StatusPasswordTransitionNeeded, from section 4.7.12.
-
454: StatusTemporaryAuthFailure, from section 4.7.0.
-
534: StatusAuthMechanismTooWeak, from section 5.7.9.
-
-
lib/mlog: make the Outf method always add new line at the end.
One of common mistakes when using logging library is to put the new line "n" at the end of format string, which cause delayed output written to Stdout (the OS wait for "n" as signal for printing).
This changes check new line to every call of Outf method and add it if its not exist.
If the caller need to call Outf multiple times before ending it with new line, they should handle it manually by storing into temporary buffer first and call Outf at the end.
-
lib/memfs: add option CommentHeader to EmbedOptions.
The CommentHeader option allow user to define custom header to the Go generated file. The string value is not checked, whether it’s a comment or not, it will rendered as is.
-
lib/ini: make the Marshal on map field sorted by keys
Given the following struct,
type ADT struct { Amap map[string]string `ini:"section:sub"` }
and ini text,
[test "map"] c = 3 b = 2 a = 1
Unmarshal-ing the text into ADT and then Marshal-ing it again will result in unpredictable keys order.
This changes fix this issue by sorting the keys on ADT.Amap on Marshal-ing, to make the written output predictable.
Bug fixes
-
lib/io: fix DirWatcher not removing old files on rename.
Previously, if a sub-directory being watched by DirWatcher is renamed, the old directory does not get removed from field dirs.
This commit fix this issue by deleting the sub directory on unmpSubdirs.
While at it, guard any read/write to dirs field with mutex to prevent data race.
-
lib/dns: check for possible index out of range when unpacking RR.
There is a possibility that record data (rdata) length inside the packet is greater than length of packet itself. Some of the reasons are corrupted packet from server or packet poisoning (attacking the DNS server by sending invalid packet).
This changes fix this issue by checking the index and rdata length with the length of packet before consuming the rdata itself.
Chores
-
lib/smtp: provide an example of how to create MailTx from email package.
If one read the current documentation on how to use the Client.SendTx, there is a missing link on how to create and populate MailTx.
This changes provide the example using the email package to generate the MailTx Data.
-
lib/mlog: change default mlog instance to non-pointer.
Since the default mlog instance is a global variable, using non-pointer give advantages on minimize GC pressure.
share v0.34.0 (2022-02-05)
Breaking changes
-
lib/sql: make the table migration customizable
In the method Migrate() we add parameter "tableMigration" which define the name of table where the state of migration will be saved.
If its empty default to "_migration".
New features
-
lib/os: implement function to Extract compressed and/or archived file
The Extract function uncompress and/or unarchive file from fileInput into directory defined by dirOutput. This is the high level API that combine standard archive/zip, archive/tar, compress/bzip2, and/or compress/gzip.
The compression and archive format is detected automatically based on the following fileInput extension:
-
.bz2: decompress using compress/bzip2.
-
.gz: decompress using compress/gzip.
-
.tar: unarchive using archive/tar.
-
.zip: unarchive using archive/zip.
-
.tar.bz2: decompress using compress/bzip2 and unarchive using archive/tar.
-
.tar.gz: decompress using compress/gzip and unarchive using archive/tar.
The output directory, dirOutput, where the decompressed and/or unarchived file stored. will be created if not exist. If its empty, it will set to current directory.
On success, the compressed and/or archived file will be removed from the file system.
-
-
lib/http: implement method Download() on Client
The Download method get a resource from remote server and write it into DownloadRequest.Output (a io.Writer).
Enhancements
-
lib/websocket: return error if parameter is empty on RegisterTextHandler
Previously, the RegisterTextHandler method return nil if method, target, or handler parameter is not set. This may cause confusion and hard to debug handler when no connection receive but the RegisterTextHandler does not have any error.
Chores
-
lib/http: change the test port for testing HTTP server
Previously, the test port for HTTP server is set to 8080 and may conflict with any service that running on the local (due to common use of 8080).
This changes it to 14832 and we make the full server address stored as global variable so any tests can references it.
share v0.33.0 (2022-01-09)
Happy New Year!
Three years has passed since the first release of this multi-libraries (or Go module), and we have released at least 33 new features with several bugs here and there.
For anyone who use this module, I hope it help you, as the module name intended "share", and sorry if its too many breaking changes.
Live long and prosper! See you again next year.
New features
-
cmd/gofilemode: new command to decode the Go file mode
The Go has their own file mode that works across all operating system. The file mode is represented by uint64, the command line will convert it to fs.FileMode and print each possible flag on it including the permission.
-
lib/sql: make the TruncateTable run with cascade and restart identity
On table that contains foreign key, truncate without cascade may cause the method fail.
Also, since TruncateTable is, and should be only, used on testing, any identity columns, for example serial, should be reset back to its initial value. On PostgreSQL this means the truncate table is with "RESTART IDENTITY".
-
cmd/epoch: command line to print and parse Unix timestamp
Program epoch print the current time (Unix seconds, milliseconds, nanoseconds, local time, and UTC time) or the time based on the epoch on first parameter. Usage,
epoch <unix-seconds|unix-milliseconds|unix-nanoseconds>
Without a parameter, it will print the current time. With single parameter, it will print the time based on that epoch.
Breaking changes
-
lib/http: refactoring NewClient to accept single struct
Previously, the NewClient function accept three parameters: serverURL, http.Header, and insecure. If we want to add another parameter, for example timeout it will cause changes on the function signature.
To prevent this changes in the future, we change it now. The NewClient now accept single struct.
While at it, we add option to set Timeout.
The Timeout affect the http Transport Timeout and TLSHandshakeTimeout. The field is optional, if not set it will set to 10 seconds.
-
lib/http: remove field memfs.Options in ServerOptions
This options is duplicate with Memfs.Opts.
-
lib/websocket: add "ok" return value on ClientManager Context
The ok return value will be true if the context exist or false otherwise.
-
lib/memfs: remove field ContentEncoding from EmbedOptions and Node
The original idea for option ContentEncoding in EmbedOptions and Node is to save spaces, compressing the content on disk on embedding and doing transport, when the MemFS instance is used to serve the (embedded) contents of file system.
This option turns out break the HTTP content negotiation [1] of accept-encoding header, if the HTTP server does not handle it properly, which default Go HTTP server does not.
In order to prevent this issue in the future, for anyone who use the memfs for serving static HTTP contents, we remove the options and store the embedded content as is and let the HTTP server handle how the compression by itself.
-
lib/email: refacforing ParseMailbox
This commit changes the signature of ParseMailbox by returning no error.
Bug fixes
-
lib/memfs: skip mount if the Root node has been initialized
-
lib/websocket: fix race conditition on handleText
Instead of accessing the ctx field directly, call the Context() method to prevent data race.
-
lib/sql: check for EOF on loadSQL
There is probably a regression in Go that cause ioutil.ReadAll return io.EOF, while it should not, because the documentation said that
A successful call returns err == nil, not err == EOF.
But in this, using http.FileSystem, the ioutil.ReadAll now return EOF and we need to check it to make the migration can run without an error.
Enhancements
-
lib/io: realign all structs
The struct realign, save the occupied of struct size in the memory,
-
DirWatcher: from 184 to 144 bytes (-40 bytes)
-
Reader: from 16 to 8 bytes (-8 bytes)
-
Watcher: from 32 to 24 bytes (-8 bytes)
-
-
lib/http: realign all structs
Changes, * Client: from 56 to 48 bytes (-8 bytes) * CORSOptions: from 104 to 88 bytes (-16 bytes) * Endpoint: from 64 to 32 bytes (-32 bytes) * EndpointRequest: from 72 to 56 bytes (-16 bytes) * route: from 56 to 32 bytes (-24 bytes)
Other changes is struct on unit tests.
-
lib/memfs: add method Init
The Init provided to initialize MemFS instance if its Options is set directly, not through New() function.
-
lib/memfs: embed the Embed options and GenFuncName
This is to make the instance of memfs initialize from init is reusable.
-
lib/memfs: realign struct Node, Options, PathNode, and on unit tests
The realign save storage spaces on struct,
-
Node: from 240 to 224 bytes (-16 bytes)
-
Options: from 112 to 104 bytes (-8 bytes)
-
PathNode: from 16 to 8 bytes (-8 bytes)
-
-
lib/email: realign the struct Mailbox
This changes the storage size from 80 to 72 bytes (-8 bytes).
Chores
-
github/workflows: remove step to get dependencies
The Go module should handle the dependencies automatically.
-
github/workflows: set go version to 1.17.6
-
lib/email: convert the unit test for ParseMailbox to examples
Since the ParseMailbox is public we can provide an examples and test at the same times.