Remote Copy with Powershell and MongoDB

Posted by claustopholt on Sep 29, 2013

At one of my recent clients, the network and Active Directory infrastructure prohibited the use of file shares between development and test environments, even for service accounts. This made some everyday things difficult — such as implementing continuous delivery mechanisms that automatically push successful builds from a TFS build farm to test environments for automated Selenium and performance testing.

Interestingly, both Powershell 2 and WinRM were considered standard components on all development and test servers (used for operations management), and so that seemed like a good place to start.

My first idea was simply to transfer the builds using Powershell remote sessions, base64 encoding the files on the sender, transmitting the file(s) as string arguments and decoding on the receiver, like so:

# Function to transfer (small) files to a PSRemoteSession using InputObject and Base64 encoding.
function PSTransfer-File
{
  param ($session, $sourcePath, $targetPath)
  
  # Get the local file and Base64 encode.
  [Byte[]]$fileBytes = [IO.File]::ReadAllBytes($sourcePath)
  $inputObject = [Convert]::ToBase64String($fileBytes)

  # Transfer.
  Invoke-Command -Session $session -InputObject $inputObject -ArgumentList @($targetPath) -ScriptBlock {
    param($targetPath)

    # Write file to target machine.
    $fileBytes = ([Convert]::FromBase64String($input))
    [IO.File]::WriteAllBytes($targetPath, $fileBytes)
  }
}

That’s a nice trick, and can be implemented from scratch in very little time*. But it suffers from a few annoying problems: The method is rather slow because base64 encoding takes up a lot of space. Also, any files larger than a few megabytes after encoding will choke WinRM. So a chunking mechanism would have to be implemented, since the build files were 20+ Mb. That’s of course possible, but it started to feel like too much of a hack.

Since our target environments had MongoDB servers available, we decided to use that instead. This had the added benefit of leaving an audit trail of all file transfers, should it ever be needed.

It’s easy enough to use the MongoDB C# driver from Powershell, you just need the Mongo.Bson.dll and Mongo.Driver.dll files and then you can upload the file(s) straight to MongoDB’s GridFS and of course also download them:

function Initialize-MongoDB
{
  param($connectionString)

  Add-Type -Path .\MongoDB.Bson.dll
  Add-Type -Path .\MongoDB.Driver.dll

  # Connect to MongoDB transport database.
  $mongoBuilder = New-Object MongoDB.Driver.MongoUrlBuilder($connectionString)
  $mongoClient = New-Object MongoDB.Driver.MongoClient($mongoBuilder.ToMongoUrl())
  $global:transportDatabase = $mongoClient.GetServer().GetDatabase($mongoBuilder.DatabaseName)
  $options = New-Object MongoDB.Driver.GridFS.MongoGridFSSettings
  $options.Root = "transport"
  $global:transportGridFS = New-Object MongoDB.Driver.GridFS.MongoGridFS($transportDatabase)
}

function Upload-FileToMongoDB
{
 param($sourcePath)

 # Upload file to GridFS.
 $fileName = [IO.Path]::GetFileName($sourcePath)
 $fileStream = New-Object System.IO.FileStream($sourcePath, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read)
 $uploadResult = $global:transportGridFS.Upload($fileStream, $fileName)
 $fileStream.Close()
 $fileStream = $null
}

function Download-FileFromMongoDB
{
 param($fileName, $targetFolder)

 # Read from MongoDB.
 $mongoFileInfo = $global:transportGridFS.FindOne($fileName)
 $readStream = $mongoFileInfo.OpenRead()
 [Byte[]]$buffer = New-Object System.Byte[] $readStream.Length
 $readBytesCount = $readStream.Read($buffer, 0, [Int32]$readStream.Length)
 $readStream.Close()
 $readStream = $null

 # Write file to disk.
 [System.IO.File]::WriteAllBytes((Join-Path $targetFolder $fileName), $buffer)
}

function Delete-FileFromMongoDB
{
 param($fileName, $targetFolder)

 # Delete file from MongoDB.
 $mongoFileInfo = $global:transportGridFS.Delete($fileName)
}

But the question was how to get the two driver dlls on to the target machines. Of course, it could be considered a requisite and installed once. But what about updates and patches? And it would be nice to be able to use this on any target machine. Simple: Start by transfering the two small MongoDB dlls using the PSTransfer-File function I described above to the target server and then use a remote session to initialize them and download the files as required.

This may seem a lot of code to simply copy a file, but it has several benefits: It’s really fast — in fact, it turned out to be much faster than xcopy on the company’s infrastructure when we tried it across various zones. It makes it possible to do auditing of who up/downloaded what, which is nice in deployment scenarios. It makes it possible to distribute files to many target servers at once. And it means the only requirement is that WinRM is active, which is the case in many corporate infrastructures because it’s used for server management.

* This by the way is also the easiest way to circumvent file security in almost any IT infrastructure.