GIT security model

Git, SHA1 and security
Is the GIT security model dependent on the cryptographic security of the hashing algorithm (SHA1) used by git to generate id's for GIT objects?
After new progresses last year in breaking the SHA1 algorithm, it is reasonable to try to find an answer to this question before deciding to adopt GIT for your software project(s). This was the subject of an interesting discussion I recently had with some colleagues.
There's an interesting post by Linus Torvalds on the  on the Cryptography Mailing List  about this subject, dated 25 Apr. 2005.
Basically it would be very difficult for an an attacker, leveraging the possibility to generate a collision in order to corrupt a GIT object database, to produce huge harms because the GIT security model is NOT based on the cryptographic security of the SHA1 hash, but on the fact that (in Linus' words)"git is distributed, which means that a developer should never actually use a public tree for his development".
And, of course, the possibility of corrupting all the existing repositories of all users involved in a project, without anybody noticing it, is quite remote.
The adoption of SHA1: a design flaw?
Even if we do not consider the adoption of SHA1 an issue by the point of view of security (i.e., we agree that that the weakness of the SHA1 algorithm does not make life easier for attackers who wants compromise the integrity of a GIT archive), still this could be considered a design flaw, since the id's for objects are not deterministically unique, but only probabilistically. My opinion? The probability of collisions of two files in a software project using SHA1 is so low that this will never be a concrete issue for GIT users (thanks to Luca Milanesio, Peter Moore and Stefano Galarraga for your input).

Jdbc, TCP channels and other funny dudes...

My most affectionate readers already know that I recently worked on a java tool that automates the migration of database structure and content between different database versions (you know, changes in tables, fields, new indexes, and other database refactoring operations): the tool is called DMT, and I already spoke about it in the pages of this blog.

The tool is used by several development teams for all operations of database re-creation, migration and tests, and I was not receiving so many complaining emails these days, so I was pretty confident that it is now decently stable and the planned tests in production-like environment would run smoothly. Of course, this is not what happened: when DMT was used in a pre-production environment for testing purposes, the DBA's who run the tests experienced that it was not able to run the migration, hanging indefinitely without completing the execution of the migration task.

This was the right occasion for me to learn what happens when a TCP channel is closed while a jdbc connection is active over it.

The problem

1. DMT connects to the Oracle host and sends through a jdbc connection the SQL statements to be executed
2. The Oracle server receives the SQL statements and begins the execution. While the SQL statements are running, there is no traffic between DMT and the Oracle server, because DMT keeps waiting from the Oracle Server a signal when the execution of the statement is completed (or when an Oracle exception occurs)
3. While the Oracle Server is running the SQL statements and DMT is waiting a signal that this process has been completed, a firewall detects that the TCP channel between DMT and the Oracle server is inactive and decides to close the connection because of a timeout configuration.
4. When Oracle completes the SQL execution, the TCP connection with DMT is no longer open, so it is not able to communicate to DMT that the job is done, and DMT keeps waiting a message forever, like an unlucky man which has an unrequited love for a woman

The solution

The TCP Keep-alive mechanism!

It's possible to enable this mechanism in a jdbc connection simply adding the parameter ENABLE=BROKEN to the jdbc string used to activate the connection, and keep_alive "probes" will be  sent over the connection after a period of inactivity keeping the connection alive. The jdbc url will look like the following one:

jdbc:oracle:thin:@(DESCRIPTION=(ENABLE=broken)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=yourhost1)(PORT=1521))(ADDRESS= (PROTOCOL=TCP)(HOST=yourhost2)(PORT=1521))(LOAD_BALANCE=on)(FAILOVER=on))(CONNECT_DATA=(SERVER=dedicated)(SERVICE_NAME=service.yourcompany.com)(FAILOVER_MODE=(TYPE=session)(METHOD=basic)(RETRIES=10)(DELAY=3))))

The TCP settings of the host OS will be used to determine when to start sending the keep-alive probes ("keep alive time" parameter), how many probes to send before detecting that the connection is closed ("keep alive probes" parameter) and the interval between two consecutive probes ("keep alive interval")(some further details here).

Bonus: What does the classic "TCP/IP Illustrated" book say about keepalive?

Many newcomers to TCP/IP are surprised to learn that no data whatsoever flows across an idle TCP connection. That is, if neither process at the ends of a TCP connection is sending data to the other, nothing is exchanged between the two TCP modules. There is no polling, for example, as you might find with other networking protocols. This means we can start a client process that establishes a TCP connection with a server, and walk away for hours, days, weeks or months, and the connection remains up. Intermediate routers can crash and reboot, phone lines may go down and back up, but as long as neither host at the ends of the connection reboots, the connection remains established . [...]

There are times, however, when a server wants to know if the client's host has either crashed and is down, or crashed and rebooted. The keepalive timer, a feature of many implementations, provides this capability.

But be aware: 

Keepalives are not part of the TCP specification. The Host Requirements RFC provides three reasons not to use them: (1) they can cause perfectly good connections to be dropped during transient failures, (2) they consume unnecessary bandwidth, and (3) they cost money on an internet that charges by the packet. Nevertheless, many implementations provide the keepalive timer. 

Hope this helps!

Exploring GIT

I passed this Sunday afternoon reading Version Control with GIT, by Jon Loeliger.
Git is the distributed version control system currently used for Linux Kernel development, conceived and developed under the protective wing of Linus Torvalds himself. The key word here is distributed: using GIT, there is no need of constant synchronization with a single, central repository, thus allowing a distributed model for software development. The book is quite interesting as it's different from most tutorials available in the web: the first chapters of the book describe the  internal data structures GIT is based on (commits, trees, blobs and tags stored in the GIT 'Object Store'), and the 'staging' mechanism implemented via the GIT 'index'; the main git commands are then explained referring systematically to these concepts, describing in detail what changes occur to the the git object store and git index as different git commands are executed. The advantage of this approach is that it forces the reader to a deeper understanding of what is behind the scenes while running each command. Of course, you'll have to spend some hours understanding these concepts before diving into git commands, but I think it's worth spending some more hours initially to properly learn a version control technology than spend a lot of hours after, running commands without a full understanding of all the implications and consequences. After all, Linus Torvalds himself stated in the GIT mailing list that you can't grasp and fully appreciate the power of GIT without understanding the purpose of the GIT index, which in turns refers to the objects in the GIT Object Store. If you are using GIT and you are not familiar with these concepts.. you should spend some time studying them, and Version Control with GIT is a good resource to have a look at.

DMT rules

I'm proud to announce that the almighty Database Management Tool (DMT) I wrote has been successfully used to migrate data to the new version of MyWeb, the mobile portal which is part of the 360 platform. DMT rocks ;-)!

Managing databases changes

Software Development is one of the most entropic fields of human activity. "Errare humanum est, but if you want to screw up everything you need a computer", as I read somewhere. Without proper management of all factors that come into play in the lifecycle of a software project, failure is the most probable epilogue of the story. Every experienced programmers know very well that delivering a software product is not simply a matter of writing code. Coding is only a part (very important, of course) of the game, but it's not the whole game, and losing the overall picture is as easy as dangerous. I am currently working on a project to manage structural databases changes and data migrations between different database versions. It's a common mistake not to think about these issues since the very first phases of design and development of software products: usually architects and developers focus on good database design, on proper Object Relational Mapping, and on performance issues. Project managers and deployers begin to think about database migrations when a new software release is dropped, which requires database changes. Is the currently deployed database already compatible with the new release, or not? How can we determine this? If changes are needed, which scripts do need to be run? Which schemas need to be changed, all of the schemas used by the application or only a subset of them? The first time I had to think about these issues, it was for a pretty huge and complex project (the Vodafone Live! portal), in a quite complicated scenario: there were localized releases of software developed on different CVS branches (localizations were needed for functional adaptations of the software for the countries that would use it), and whenever a new release was dropped deployers needed to know how to properly migrate the database and how to change the schemas so that it could work with the new version. Finally, after talking a bit about this with some colleagues (mainly Peter Moore, Mida Boghetich, and Alexis Konstantopoulos, thank you guys) I came out with some basic principles to approach this issue.

The most important thing was to define a good database versioning mechanism. When I say "database version" I mean a unique identifier that describes the full set of Users and, for each User, Schema Objects (tables, constraints, indexes, stored procedures) that are required by given version(s) of the software. We choose to name database versions using the CVS labels used to tag the software modules responsible of the database creation. Then we started re-organizing the existing database migration scripts (mainly sql scripts) as point to point migrations, that means migrations from a database version to another. (The database version needs to be a separated, independent concept from the software version, because different software versions can possibly require the same database version; consequently, not all releases do imply the need of a database migration). At this point we needed to solve the following problem: how can we check that a given database installation is compliant with a given database version? In other words, how can we run a database compliance test on an existing database instance, to see if the database structure is compliant with a given db version or it has been corrupted, or not properly installed, or a past migration has not been run properly? For each database version, a textual description of all users was generated in a definite format, and to do that a dedicated tool was written which was able to connect to an existing database instance and generate the desired textual description of the database, so that comparing the actual database structure with the expected one was as simple as diffing two text files (The tool simply queries the database metadata tables to get all the required information, and formats the result according to a definite set of rules). The last ingredient of the recipe was a simple algorithm (let's call it the pathfinder algorithm) to compose existing atomic migrations in order to run generic ones. For example, let's imagine that the following database migrations have been written: A to B, B to C, and C to D. If we need to migrate from A to D, we will need to run these three migrations in the correct order; this is precisely the pathfinder's job: we tell the pathfinder which migration we need to run (Version Alfa to Version Omega), and it will tell us which existing migrations we need to run, and in which order (i.e., Alfa to X, X to Y, Y to F34, F34 to H67, H67 to Omega). Of course the provided examples are pretty simple, but you can easily imagine that if many database versions need to be supported, it's convenient to have a dedicated tool to compute which scripts need to be run and in which order.

To sum up, the solution that was successfully used to properly manage database changes was made up of the following "ingredients":

1. A Database versioning mechanism: for each software release it must be clear what is the required database version, and this must be identified univocally with a string identifier
2. Point to Point migrations: all sql scripts to manage schema changes and data migrations need to be written so that it's perfectly clear what the source database version and the target database version are.
3. A pathfinder tool, able to compose the existing point to point migrations into generic ones
4. Compliance tests, to automatically check if a database installation is compliant with a given database version

The previous description is just a short introduction to the approach I am currently using to manage database changes. Actually, I am working on a simple java tool to automate many of the processes implied by the adoption of the described key concepts. Furthermore, there are several existing tools and resources that can be very useful to handle database changes, or simply to provide inspiration regarding these topics: www.liquibase.org, www.dbunit.org; I'd recommend also the "migrations" chapter in the Doctrine Manual (http://www.doctrine-project.org).