# Side-by-side installation of Cygwin SSH and Git on Windows

I have a Cygwin installation on my Windows box and now I downloaded the lastest Git version (for Windows). Git has a small Cygwin included as well and I thought the two would interact nicely—but they don’t.

The problem is that they use a different layout of mapped drives (i.e., mapped from Windows to the Posix world of Cygwin). My regular Cygwin has its root under C:\cygwin64 and from there, for example, the SSH key directory is under /home/chris/.ssh, whereas Git has its root directory somewhere in the Git installation and the home directory is where the Windows user directory is, i.e., C:\Users\chris and there lies the .ssh directory as well.

I thought some symbolic links would do the trick, so I linked the Git .ssh directory to my (real) .ssh directory in Cygwin.

Nice, but not enough. Now, SSH will not find the keys, since it still wants to load its keys from /c/Users/chris/.ssh, so I added another symbolic link to add the /home directory to Git and map it to the /home of Cygwin and add a symbolic link to /etc/ssh_config and add an IdentityFile /home/chris/.ssh/id_rsa.

Not bad, but still not good enough. This time, Git’s ssh-agent is not able to connect to the already loaded (regular Cygwin) ssh-agent. The reason is that both Cygwin’s have different /tmp directories. When I added the Git version to my regular Cygwin (that is, have /tmp map to the user’s TEMP directory, usually under C:\Users\chris\AppData\Local\Temp) by changing the /etc/fstab file, nothing changed. Calling mount to check if the changes worked showed no changes.

The trick here is: you have to close all programs that use Cygwin: ssh-agent, bash, sshd, mintty, everything. Then reload and Voilà! Now the proper /tmp is there and it’s the same on both Cygwin’s, ssh-agent can be accessed from both installations and due to the symbolic links both SSH’s use the same config and key files.

Victory at last!

# I use cFosSpeed to keep my Ping time nice and low

I use cFosSpeed to keep my Ping time low and be able to use VoIP, terminal sessions, remote desktop, IRC sessions and online games on the one hand and up/downloads on the other hand at the same time. cFosSpeed keeps the “pressure” of the file transfers on the line low, so urgent data can still pass quickly.

Since I’m one of the developers of cFosSpeed it’s probably no wonder I like it and endorse its use.

So, if you’re interested, click here to get to the cFosSpeed site:

# PtCut 3.0.1 and PPLpy for Python 2.7 wheels released

Jonas Weinz has produced pre-compiled wheels of PPLpy  for Python 2.7 (for Linux 64-bit). Have look at Jonas’ page here.

I have mirrored the files here as well:

I have released a slightly updated version 3.0.1 of PtCut that works under plain Python 2.7 as well (and Python 3.x and SageMath).

# Simple Top-Down Parsing in Python 3.x

I found this nice top-down parser on Fredrik Lundh’s site http://effbot.org/ (thanks, Fredrik!), but unfortunately, it was in Python 2.x.  To make it run under Python 3.x, you only need some small changes, but where took me some time…

# Getting a kernel mode driver signed for Windows 10

In this article I want to describe my experiences with the new (as of August 2016) driver signing issues and Windows 10.

Since the Anniversary Update of Windows 10 (version 1607, also called Redstone1), Microsoft requires new signatures on your kernel mode drivers under certain circumstances. This is called “attestation signing”. That is to say, only if certain circumstances are met — and I guess these “loopholes” will get smaller over the years — you can run drivers that were signed the “old” way (called “cross-certification”) on Windows 10.

The “new” way brings certain big changes:

• The “old” way meant you — as the software developer — use your certificate to sign your software. Since your certificate is cross-signed by a certification authority (CA) that is (in the end) trusted by Microsoft this ensures that no one could tamper with the file and that Microsoft trusts that you are who you claim you are.
• The “new” way means that you submit your software to Microsoft and they add their certificate, provided all the requirements are met.
• For that, you need an EV code signing certificate, there is no way around it. This basically means, it’s more expensive and it comes on an USB hardware token (so you cannot copy it).

To give an idea of what has to be done, I describe what I did. My company, cFos Software GmbH, needs cFosSpeed, our traffic shaping driver signed for Windows 10.

### Getting an EV certificate

I got the certificate from Globalsign Germany, as I always did, just this time I ordered the “extended verification” (EV) certificate. This increased the price from 429€ (about $480 as of 2016-09-02) for 3 years to 709€ (about$793)! (Note to self: in next life, run a certification authority!)

The process is kinda smooth, they require some (updated) documents (HRB printout), check that the provided contact data (email address and phone number) is correct and have you sign the agreement. They were very helpful and the whole process required some three days plus another two until I got the USB token by mail. If this is your first time you order a certificate from a CA, it might take longer, since they must check more data.

On that USB token you have to download the certificate from their website. You can do that only once, so there is no easy way to share the same certificate with several people (two teams in two offices, for example). You cannot just copy the USB token, of course.

The actual signing works as always, only now a password box pops up every time you sign. Luckily, copy-and-paste works for the password entry.

On the plus side, Microsoft SmartScreen instantly trusts files that where signed with an EV certificate. So that might ease out the roll-out of new software a bit.

Here is what Microsoft says about getting a code signing certificate.

### Getting the Microsoft Signature

##### Prerequisites

It’s harder as I imagined (of course). Firstly, you need a Microsoft sysdev account. I don’t recall how I created that, but I gather it wasn’t that hard.

Secondly, you need to download some file (called winqual.exe)  from Microsoft (under Administration / manage certificates), sign it with your EV certificate and upload it again. This way, you only prove that your company has an EV certificate. That same certificate has to be used later to sign your submission.

Thirdly, you need to “sign” some legal documents. “Sign” luckily only means typing your name and the date and it’s almost instantly countersigned. They don’t say which documents need signing beforehand, so I guessed we might need “Windows Compatibility Program and Driver Quality Attestment Testing Agreement” and signed it. But that wasn’t enough, so I signed “Windows Certification Program Testing Agreement v1.0” as well, which seemed to be almost the same. After that, I was able to proceed further.

##### Submitting

Now I was able to upload our driver for signing. There are two ways of submitting:

• Using the Hardware Lab Kit (HLK) to test your submission against Windows 10 and use the Hardware Certification Kit (HCK) to test against earlier versions of Windows. Merge the results and upload that to Microsoft.
• Cross-sign the drivers yourself and upload to Microsoft for attestation signing. That is what I did. Microsoft mentions that drivers signed this way won’t run on Windows Server 2016 Technical Preview.

Firstly, I sign the files myself in the usual way using signtool.exe:

signtool sign /a /ac GlobalSign_Root_CA.crt /s my /n “Company name” /fd sha256 /td sha256 /tr “http://timestamp.globalsign.com/?signature=sha2” /du “http://www.example.com” cabfilename.cab

You need the “sha256” options, since SHA1 has been deprecated since 2016-01-01 and will work less and less in the future. Make sure that you use the same certificate you used to sign winqual.exe earlier on.

Secondly, you need to put the .SYS driver file and its .INF file into one .CAB archive into a subdirectory. Here is a description from Microsoft on how to actually pack the files and sign them by yourself. Note, that I used CABARC instead of MAKECAB to pack the files:

md driver
copy driver.sys driver.inf driver
cabarc -p -r N cabfilename.cab driver\*

I chose to include our .CAT file as well, even though the submission process will create one for you. I don’t know yet if that was a good or bad choice, both ways seem to work.

Thirdly, sign that .CAB file like you signed the .SYS file (see above).

Forthly, I uploaded that signed .CAB archive to sysdev.microsoft.com. You will have to choose for which versions of Windows 10 your driver qualifies. I checked all versions, but only one architecture (x64 or x86). You can only have one architecture per submission!

### … and waiting

The submission takes a while and goes through a ten step process, namely:

1. (no idea)
2. Transferring CAB File
3. Scanning CAB file for Viruses
4. Decompressing CAB File
5. Validating HCK/HLK Submission Package
6. Creating Catalog Files
7. Archiving Files
8. Parsing Driver Data
9. Signing Catalog Files
10. Transferring Catalog File to Server

OSR describes in this post that the whole process took them 30 Minutes. But the first time I submitted, step 5 took six days and hadn’t completed! Then I wrote an email to sysdev@microsoft.com to find out that I had built my .CAB file the wrong way (I had put all files into the root folder instead of a “driver” sub-folder) and that had apparantly hung the process. (Thanks again for a timely and succinct answer, Jack!) So If you have to wait for long period of time, it’s probably best to contact Microsoft by mail and solve the issue.

Fixing that problem and re-submitting the driver package got me an approved driver within 10 minutes! 🙂 🙂

### Victory!

After that you can download the signed .SYS and .CAT files packed into a .ZIP file. When you install that driver it doesn’t pop up that box asking “Do you trust Company X?” So, that’s nice.

What’s less nice is that drivers signed like this, even though they retain our old EV signature, don’t load under Windows 8.1 and load like they have no signature at all (and thus look kinda bogus, since they have no reference to the publisher) under Windows 7. So far, I have found no way to have a single driver file that loads under all Windows 7+ operating systems.

# Curve-fitting With Minimized Relative Error

### The Problem

I wrote a C++ function to multiply two large positive integers of the same length, say $$n$$ 64-bit words, with the grade-school method. Let’s call that function omul_n(). Then, I wrote extensive benchmarking to assess the speed of my efforts. The resulting run-times for the multiplication of two numbers with $$n$$ words look like this:

WordsCycles
118
9281
17915
251959
333421
415207
497392
5710093
6513000
7316397
8120224
8924326
9728800
10533941
11339764
12145487
12951212
13757453
14564142
15371778

Now I wanted to find a closed function to most accurately describe the run-time of omul_n() We know that to multiply two numbers of $$n$$ digits each, we need to do $$n^2$$ digit-multiplications. So, most likely, the desired function will look something like $$T(n) = c_0 + c_1 n + c_2 n^2.$$

The only question is: what values to use for $$c_0$$, $$c_1$$ and $$c_2$$? I like linear regression, but it only works for linear relationships, like $$T(n) = c_0 + c_1 n$$. We cannot use that here.

### The First Solution

The solution to my question is curve-fitting. I used Python functions to do so, namely scipy.optimize.curve_fit from the SciPy package (a good starter article that inspired my use of curve-fittings is here.)

The program is really simple. You input your data plus the describing function (like $$T(n)$$ above) into the curve-fitting function and out pop the coefficients $$c_i$$ that yield the $$T(n)$$ with the least squared error.

The Python script:

omul_str = open("omul-speed.txt", "r").read() # read measured values
o = [float(i) for i in omul_str.split()] # make one big list
os = o[0::2]                             # slice out first column
ot = o[1::2]                             # slice out second column

import numpy as np                       # imports
from scipy.optimize import curve_fit     # the magic function

xdata = np.array(os)                     # convert lists to np.array
ydata = np.array(ot)
def func(x, c0, c1, c2):                 # the modeled function
return c0 + c1*x + c2*x*x

popt, pcov = curve_fit(func, xdata, ydata) # and fit it!
print(popt)                              # print optimized parameters

If you’re not used to NumPy, array features an unfamiliar usage:

Python 3.4.1 |Anaconda 2.1.0 (64-bit)| ...
>>> import numpy as np
>>> a = np.array([1,2,3])
>>> a
array([1, 2, 3])
>>> import math
>>> math.log(a)
Traceback (most recent call last):
File "", line 1, in
TypeError: only length-1 arrays can be converted to Python scalars
>>> np.log(a)
array([ 0.        ,  0.69314718,  1.09861229])


NumPy functions that are applied to an array again return an array with values of said function applied to every array element. That comes in pretty handy when handling larger sets of data.

Back to our curve-fitting. The above listed script generates this output:

[-60.37910437   5.09798716   3.03566267]

That means that the best fitting function is about
$$T_\text{abs}(n) = -60 + 5.1 \cdot n + 3.04 \cdot n^2.$$

Pretty neat, eh? Plotted it looks like this. The red line is not the connection of the dots, but our model:

### My Discontent

So far, so very cool. An issue arises when we look at the relative errors between data points and model. That is, $$|T(n) / T_n|$$, where $$T(n)$$ is our model and $$T_n$$ is the measured run-time. In contrast, the above curve-fitting minimized the absolute error $$|T(n) – T_n|$$. (Actually, it minimized the squared absolute error, but I let that slide here and focus on absolute vs. relative error.)

Some additional lines of Python code added to the end of our script will print the relative errors and their average:

relerr = abs(1 - ydata / func(xdata, *popt))    # relative errors
np.set_printoptions(suppress=True)              # switch off sci. notation
print(relerr * 100)
avgrel = sum(relerr) / len(ydata) * 100         # calc average
print("avgrel:", avgrel)


Which does produce this extra output:

[ 134.45275796   21.43922899    1.26238363    0.27284922    0.21410507
0.84902538    1.15067892    0.00073479    0.73808731    0.55686398
0.22467506    0.46166589    0.6782697     0.00615863    1.23715341
1.07859592    0.19226999    0.28013432    0.56064524    0.00479269]
avgrel: 8.28305380503


So, we have an average relative error of 8 %, which seems rather high for me. Obviously, the relative error is extremely high with the two starting values: 134 % and 21 %. Can we improve that? That is, can we model so that the average and maximum relative error is lower?

### The Improved Solution

Least squares optimization with minimized absolute error is used very widely, but unfortunately, there is no easy way to switch the functions performing this to minimize the relative error. But I found this forum post that was very helpful. It’s on some other math software system, but we can borrow the idea: “Usually the best way to do relative error is to log your model. This changes a proportional error structure into an additive one, which is exactly what you want” (with “log” as in
logarithm).

Luckily, that is very easy to accomplish in Python. This is a changed version of the earlier script:

omul_str = open("omul-speed.txt", "r").read()   # read measured values
o = [float(i) for i in omul_str.split()]        # make one big list
os = o[0::2]                                    # slice out first column
ot = o[1::2]                                    # slice out second column

import numpy as np                              # imports
from scipy.optimize import curve_fit            # the magic function

xdata = np.array(os)                            # convert lists to np.array
ydata = np.array(ot)
def func(x, c0, c1, c2):                        # the modeled function
return c0 + c1*x + c2*x*x
def logfunc(x, c0, c1, c2):                     # ... and the log of it
return np.log(func(x, c0, c1, c2))

popt, pcov = curve_fit(logfunc, xdata, np.log(ydata))  # and fit it!
print(popt)                                     # print optimized parameters

relerr = abs(1 - ydata / func(xdata, *popt))    # relative errors
np.set_printoptions(suppress=True)              # switch off sci. notation
print(relerr * 100)
avgrel = sum(relerr) / len(ydata) * 100         # calc average
print("avgrel:", avgrel)


And now the output looks like this:

[ 12.98237958   1.9705695    3.05332744]
[ 0.03485745  1.06567529  1.49572472  0.58745607  0.52644119  0.37155771
0.65289914  0.47219135  0.31728127  0.18879885  0.09165894  0.19598836
0.45928895  0.171688    1.37775523  1.18298158  0.26311364  0.23936649
0.54721279  0.01646704]
avgrel: 0.512920201737


Awesome! The average relative error is down to 0.5 % with a maximum of 1.5 %.

The linear plot looks largely the same, because the absolute differences are too small to see. But if we switch to a double-logarithmic plot, we can see them clearly:

Clearly, the smaller the values are, the larger the difference is between the red graph (minimized absolute errors model) and the data points, whereas the green graph (minimized relative errors) is much closer to the data points for small $$n$$.