Hello,

On Thu, Nov 04, 2021 at 01:47:07AM +0000, YruNlsr6371bDg wrote:
> Ok, so we finally got around to building 'megatools-1.11.0-git-20210505.tar.gz'
> on Windows msys2/mingw64-gcc.
> 
> By the way, it was our first time using meson/ninja and it was surprisingly much
> easier to get working than using GNU autotools, which from our previous
> experience, was very error-prone on Windows cygwin64-gcc or msys2/mingw64-gcc.
> 
> We continued to face that final error message (after 100.0% uploaded) - and so,
> as did exactly like you suggested - inserted a do-while loop to re-do that final
> http_post until it succeeded - and inserted a g_printerr, g_usleep statements in
> the loop for debugging and non-DDoS respectively.

Please also look at:

  https://github.com/megous/megatools/issues/462

Looks like you may have the same issue.

The fix is already in git and the last experimental builds.
  https://megous.com/git/megatools/commit/?id=4adc57ff3c95868607510e1f4fcb08fd76cfd37b

Myabe it will help even over tor.

kind regards,
	o.

> This seemed to fix the problem - rarely - about 1 out of 5 times - the first
> http_post works to finalize/complete the upload, but with this do-while loop
> hack, the second http_post works to finalize/complete the upload - about 4 out
> of the remaining 5 times.
> 
> Again, like you said, this bug only seems to show itself when using 'tor' - and
> we confirmed that it happens due to using 'tor'.
> 
> We also use 'obfs4proxy' to obfuscate our 'tor' connection and prevent
> ISP/middleman from knowing we are using 'tor'.
> 
> And prior to fixing this bug with this do-while loop hack, we tried both with
> and without using 'obfs4proxy' - and the bug remained exactly the the same
> whether or not 'obfs4proxy' was used - so, it has nothing to do with
> 'obfs4proxy' but exclusively to do with 'tor'.
> 
> As to why this do-while loop hack was needed to fix this bug after many, many
> months of successfully (without any error) uploading to our megatools account
> using 'tor' without this hack, we believe that one of the following is true:
> 
> (1) unlikely: Either ISP/middleman does not like the bandwidth we are using by
> uploading large files which takes a long time (especially over tor) - and so,
> they are sabotaging the final connection.
> 
> (2) maybe: The 'tor' exit nodes do not like tor-users that are using up
> limited-bandwidth for uploading large files to mega.nz for long periods of time
> - and so, they are dropping the final connection.
> 
> (3) likely: mega.nz is trying to curb 'tor' usage or limit the
> amount/size/volume of uploads from 'tor' connections, and so, they have
> introduced this final block to deter/discourage tor users.
> 
> We don't use 'tor' for no reason - 'tor' users include journalists,
> whistleblowers, activists/protesters, political dissidents, oppressed
> minorities, military/top-secret-government-employeees - and so, all of these
> people need to use 'tor' as a legitimate method to avoid potentially-dangerous
> surveillance that might be harmful to them if their enemies knew what they were
> doing online.
> 
> 
> Instead of uploading a patch to you, we will simply paste as follows, a GNU
> 'sed' command that patches the code to hack this bug - the patched file is
> 'lib/mega.c' and the original file is 'lib/mega.c.sed' so, that you can compare
> the two:
> 
> /usr/bin/sed --in-place=.sed 's/ GString \*res_str = http_post(s->http, url,
> req_node, strlen(req_node), &local_err);/ GString\* res_str = NULL;\n gint
> number_of_httppost_retries = 0;\n do {\n local_err = NULL;\n res_str =
> http_post(s->http, url, req_node, strlen(req_node), \&local_err);\n
> ++number_of_httppost_retries;\n if (number_of_httppost_retries > 1) {
> g_printerr("ERROR: Having to re-do http_post due to error: %d times...\\n",
> number_of_httppost_retries); }\n g_usleep(250000);\n } while (res_str ==
> NULL);\n/g' ./lib/mega.c
> 
> 
> This hack patch should only modify 1 line into 10 lines:
> 
> $ /usr/bin/diff --report-identical-files ./lib/mega.c ./lib/mega.c.sed
> 1417,1426c1417
> < GString* res_str = NULL;
> < gint number_of_httppost_retries = 0;
> < do {
> < local_err = NULL;
> < res_str = http_post(s->http, url, req_node, strlen(req_node), &local_err);
> < ++number_of_httppost_retries;
> < if (number_of_httppost_retries > 1) { g_printerr("ERROR: Having to re-do
> http_post due to error: %d times...\n", number_of_httppost_retries); }
> < g_usleep(250000);
> < } while (res_str == NULL);
> < 
> ---
> > GString *res_str = http_post(s->http, url, req_node, strlen(req_node),
> &local_err);
> 
> 
> 
> 
> Thanks!
> 
> 
> 
> 
> 
> > On Monday, 18. October 2021 2:56, Ondřej Jirman
> [/webmail/send?to=megatools@megous.com] wrote:
> 
> Hi,
> 
> On Thu, Oct 07, 2021 at 04:46:52AM +0000, YruNlsr6371bDg wrote:
> > Hello,
> >
> >
> >
> > We tried the following using 'megatools-1.11.0-git-20210505-win64' (but we
> > presume the same error message will apply to all recent megatools versions
> > across all platforms).
> >
> > We run 'megaput' with the following command-line arguments to increase
> > verbosity/debugging:
> >
> > megatools.exe put --ignore-config-file --no-ask-password
> > --debug=http,api,fs,cache,tman --proxy=socks5h://127.0.0.1:9050
> > --username=************************ --password=***************************
> >
> '************************************************************************************************************************************'
> >
> >
> >
> > After many months of successfully using megatools to access/upload to our Mega
> > account, since approximately 5 days ago, we are now not able to complete
> upload
> > larger files to our Mega account using megatools due to end-of-upload error:
> > "error:1421C0F8:SSL routines:set_client_ciphersuite:unknown cipher returned"
> >
> > (We also tried deleting the cache-files/temporary-directory, but that did not
> > fix the problem either.)
> >
> > This issue might be due to a change in the way Mega servers handle the upload
> of
> > large files - and thus, the HTTP post fails due to the the initial connection
> > being too old ("Too old connection (***** seconds), disconnect it"):
> >
> > If so, the solution might be to keep the original connection alive - even if
> it
> > is a long upload, so that reconnection is not required - this is probably what
> > results in the failed upload, or at least provide a command-line option to
> keep
> > keep the original connection alive such as "--keep-alive" similar to the
> OpenSSH
> > "TCPKeepAlive" option.
> >
> > We also looked at the source code from
> >
> 'https://megatools.megous.com/builds/experimental/megatools-1.11.0-git-20210505.tar.gz'
> > to try to debug this problem, but this seems to be a curl library error caused
> > by disconnecting/reconnecting the "too old" original conneciton - which the
> Mega
> > servers might not be allowing any more.
> >
> >
> > The final error message received after reaching 100.0% is uploaded is posted
> as
> > follows (with any private information redacted).
> >
> > In all cases, even when we repeatedly try to upload large files, the upload
> > correctly progresses from 0.00% to 100.00% - but when reaching 100.00%, it
> > returns the following text within quotes:
> 
> Mega is likely just not caring for connection to API server. HTTP is stateless
> protocol. It's weird that connection fails, but give that this is over tor,
> it may not be mega.nz failing but something in between.
> 
> Maybe re-trying the API calls in case of this kind of failure may help.
> 
> I'd suggest you try to modify the code to add a bunch of retries to the final
> API call that finalizes the upload.
> 
> kind regards,
> o.
> 
> > "
> > .
> > .
> > .
> >
> *********************************************************************************************************************************:
> > 100.00% - done *.* GiB (avg. ***.* KiB/s)
> > 1 left intact
> > * Found bundle for host **********.userstorage.mega.co.nz: 0x***********
> > [serially]
> > * Can not multiplex, even if we wanted to!
> > * Re-using existing connection! (#0) with proxy 127.0.0.1
> > * Connected to 127.0.0.1 (127.0.0.1) port 9050 (#0)
> > > POST
> >
> /ul/**************************************************************************************/**********?c=****************
> > HTTP/1.1
> > Host: **********.userstorage.mega.co.nz
> > Accept: */*
> > Accept-Encoding: deflate, gzip
> > User-Agent: Megatools (1.11.0)
> > Content-Type: application/octet-stream
> > Content-Length: *******
> >
> > * We are completely uploaded and fine
> > * Mark bundle as not supporting multiuse
> > < HTTP/1.1 200 OK
> > < Content-Type: text/plain
> > < Content-Length: *
> > < Access-Control-Allow-Origin: *
> > < Access-Control-Allow-Headers: MEGA-Chrome-Antileak
> > < Access-Control-Max-Age: 86400
> > < Cache-Control: no-store
> > <
> > * Connection #0 to host 127.0.0.1 left intact
> > * Found bundle for host **********.userstorage.mega.co.nz: 0x***********
> > [serially]
> > * Can not multiplex, even if we wanted to!
> > * Re-using existing connection! (#0) with proxy 127.0.0.1
> > * Connected to 127.0.0.1 (127.0.0.1) port 9050 (#0)
> > > POST
> >
> /ul/**************************************************************************************/**********?c=****************
> > HTTP/1.1
> > Host: **********.userstorage.mega.co.nz
> > Accept: */*
> > Accept-Encoding: deflate, gzip
> > User-Agent: Megatools (1.11.0)
> > Content-Type: application/octet-stream
> > Content-Length: ******
> >
> > * We are completely uploaded and fine
> > * Mark bundle as not supporting multiuse
> > < HTTP/1.1 200 OK
> > < Content-Type: text/plain
> > < Content-Length: *
> > < Access-Control-Allow-Origin: *
> > < Access-Control-Allow-Headers: MEGA-Chrome-Antileak
> > < Access-Control-Max-Age: 86400
> > < Cache-Control: no-store
> > <
> > * Connection #0 to host 127.0.0.1 left intact
> > * Mark bundle as not supporting multiuse
> > < HTTP/1.1 200 OK
> > < Content-Type: text/plain
> > < Content-Length: *
> > < Access-Control-Allow-Origin: *
> > < Access-Control-Allow-Headers: MEGA-Chrome-Antileak
> > < Access-Control-Max-Age: 86400
> > < Cache-Control: no-store
> > <
> > * Connection #0 to host 127.0.0.1 left intact
> > * Mark bundle as not supporting multiuse
> > < HTTP/1.1 200 OK
> > < Content-Type: text/plain
> > < Content-Length: *
> > < Access-Control-Allow-Origin: *
> > < Access-Control-Allow-Headers: MEGA-Chrome-Antileak
> > < Access-Control-Max-Age: 86400
> > < Cache-Control: no-store
> > <
> > * Connection #0 to host 127.0.0.1 left intact
> > * Mark bundle as not supporting multiuse
> > < HTTP/1.1 200 OK
> > < Content-Type: text/plain
> > < Content-Length: *
> > < Access-Control-Allow-Origin: *
> > < Access-Control-Allow-Headers: MEGA-Chrome-Antileak
> > < Access-Control-Max-Age: 86400
> > < Cache-Control: no-store
> > <
> > * Connection #0 to host 127.0.0.1 left intact
> > * Mark bundle as not supporting multiuse
> > < HTTP/1.1 200 OK
> > < Content-Length: **
> > < X-Content-Length: **********
> > < Content-Type: application/octet-stream
> > < Access-Control-Allow-Origin: *
> > < Access-Control-Allow-Headers: MEGA-Chrome-Antileak
> > < Access-Control-Max-Age: 86400
> > < Cache-Control: no-store
> > <
> > * Connection #0 to host 127.0.0.1 left intact
> > -> [{
> > "a": "p",
> > "t": "********",
> > "n": [{
> > "h": "************************************************",
> > "t": 0,
> > "k": "*******************************************",
> > "a":
> >
> "************************************************************************************************************************************************************************************************",
> > "fa": null
> > }]
> > }]
> > * Too old connection (***** seconds), disconnect it
> > * Connection 0 seems to be dead!
> > * Closing connection 0
> > * Trying 127.0.0.1:9050...
> > * SOCKS5 connect to g.api.mega.co.nz:443 (remotely resolved)
> > * SOCKS5 request granted.
> > * Connected to 127.0.0.1 (127.0.0.1) port 9050 (#1)
> > * ALPN, offering h2
> > * ALPN, offering http/1.1
> > * SSL re-using session ID
> > * error:1421C0F8:SSL routines:set_client_ciphersuite:unknown cipher returned
> > * Closing connection 1
> > ERROR: Upload failed for
> >
> '************************************************************************************************************************************':
> > API call 'p' failed: HTTP POST failed: CURL error: SSL connect error
> > SAVE CACHE: {
> > "version": 4,
> > "last_refresh": **********,
> > .
> > .
> > .
> > "