31Dec 2019

How to Develop an Object Vault Database?

No-SQL databases are very popular nowadays. In that context, no-SQL object databases are becoming more and more of interest. They allow manipulating directly an object from and to a database without having to use complicated drivers.

In classical application development, to manipulate objects within databases, one must consider developing several layers:

  • An object repository specific to the object, usually allowing CRUD operations search
  • If this is a SQL database, the requests of the object repository must be translated into the SQL language. If this is a no-SQL database the object repository must translate the calls into the no-SQL API
  • A driver such as an odbc (C,C++,C#) or jdbc (java) driver, for instance, to send the SQL requests to the database, or for instance the MongoDB drivers.

To avoid such a hassle, it is possible to use Object databases.

Overview of Object Databases

Overview of Object Databases

Object databases are not a brand new concept. They have been developed since the ’80s but their usage in the “general” development project is very recent.

Object databases can store directly Objects and understand object-oriented programming in various ways. 

Object databases are relatively confidential systems used in niche applications like GIS, engineering, telecommunications, etc…

The main Object databases are:

Several of these databases understand the Object Query Language (OQL).

There exist additionally “light” object databases for C# like NDatabase or velocityDB.

In what follows we will build a small, light, simple but efficient object database for C# – in fact, a data vault –  which will fit any application which requires but moderate access to the database.

Our data vault will not be optimized but on many occasions, it will fit for application who do not need at all the sophisticated features of the SQL or no-SQL database engines.

Imagine an application for a private network of a hundred people like for example a group of lawyers or a small company. The application will make but only moderate use of data. The machine they will use will be equipped with modern and fast CPUs. Do they really need to gain 5 ms on the operations? Probably not. They need something secure and quickly functional.

Definition of the Vault

Our vault must be able to store any C# object, to retrieve it, to modify it and to run queries in the vault to retrieve objects.

We will not implement ACID (Atomicity, Consistency, Isolation, Durability) in our vault. This is a demo project only aimed at demonstrating some of the features of the .NET language. Anyway, that object vault can be used with standard development projects and should perform relatively well.

We will also not implement a remote protocol. This will be a flat database. In fact, we will not even store the objects into a set of a file but inside a file structure that will directly represent the objects.

Our vault will be implemented as a secure vault. Ideally using a ciphering system so that each object is stored in a ciphered way, natively, by default.

The analogy can be done with safes in a bank, individually locked and well ordered.

A lock file will be maintained so that the object can be securely accessed. 

When a vault client access an object, a lock is created which prevents other clients to modify it. When the lock is released, the object becomes available again for other vault clients.

There will not be “tables” but folders and each object type will have their own dedicated folders,

Our vault will never delete physically objects they will be “marked” as deleted.

At this stage, we will not develop the ability to create tables which could mix different types of objects with eventually a query language, but this could be done, without a lot of difficulties, as an extension of the project.

In fact, the object storage can delegate to the developers the task of maintaining tables of objects with indexes of object IDs and retrieving the objects using the vault. The same queries, like LINQ queries, can be done outside of the vault area.

The vault will have the functionalities of an “object storage” system.

Here are the operations that our object database will be able to perform:

NameDescription
deleteObject<T>(string, DataVaultKey)delete an existing object of type T in the vault. No objects are deleted in the vault, they are ‘marked’ as deleted
getAllObjects<T>(DataVaultKey)get ALL the objects of type T(should never be allowed)
getCount<T>()Get total number of records for an object
getIndexFromVaultID<T>(string)get the record number from a given vault ID
getObject<T>(int, int, DataVaultKey)Returns a list of transactions as stored in the vault
getObject<T>(int, DataVaultKey)get an object from a record number
getObjectFromVaultID<T>(string, DataVaultKey)get an object from the vault using the vault id
insertObject<T>(T, DataVaultKey)insert a new object of type T in the vault
updateObject<T>(T, DataVaultKey)update an existing object of type T in the vault

The DataVaultKey is a structure containing credentials that allow access to the vault object.

Our objects will also have to implement a dataVaultStorage class so that they get a unique storage ID.

Storing the Objects

Storing the Objects

In order to store and manipulate the objects, we shall simply use a serializer. There are many ways it can be achieved. For this, we will implement a SerializerFactory.

The factory will produce a serializer for a given type T of the object with the following code:

public static  XmlSerializer getSerializer<T>()
    {

          if(!dic.ContainsKey(typeof(T)))
          dic[typeof(T)] = new XmlSerializer(typeof(T));

          return dic[typeof(T)];

    }

We use a dictionary containing well-known serializers for some types, as a basic memoization technique.

When started the datavault will create unless it already exists, a root directory which will be the base for the vault.

In this root directory, objects will be stored in dedicated folders. In each folder, an idx file will maintain a list of the objects.

For instance, if we store 100 objects from an Employee class.

 DataVaultKey key = new DataVaultKey();
            key.hashKey = "FFFF";
            key.password = "1234";
            key.user = "admin";

            for (int i = 0; i < 100; i++)
            {

                Employee roger = new Employee();
                roger.Age = 23+i%55;
                roger.Name = "Roger"+i;


                DataVault.insertObject<Employee>(roger, key);
            }

We will get the following file structure:

An index file named “ idx” will be created and will store the correspondence between the objects and their storage ID.

Implementation

The following code will insert an object into the vault:

/// <summary>
        /// insert a new object of type T in the vault
        /// 
        /// </summary>
        /// <param name="new_Transaction">object to insert</param>
        /// <param name="key">the vault credentials</param>
        /// <returns>true if object inserted , false otherwise</returns>
        public static bool insertObject<T>(T new_Transaction, DataVaultKey key) where T : dataVaultStorage 
        {
            Contract.Assume(new_Transaction != null);

            Dictionary<int, string> index = null;

            //lock the index file

            try
            {
                int cticks = 0;

                while ((index == null) &amp;&amp; (cticks < 100))
                {
                    index = getIndexAndLock(typeof(T).Name);
                    System.Threading.Thread.Sleep(100);
                    cticks++;
                }

                if (index == null)
                {

                    Tracer.Trace(Tracer.severity.WARNING, "cannot acquire lock on index file");
                    return false;
                }

                //insert the data

                int c = index.Count;

                int d = c + 1;

                Guid id = Guid.NewGuid();

                new_Transaction.vault_id = id;
                
                string data = cipherObject<T>(new_Transaction, key);

                File.Create(VAULT_PATH + "\\" + typeof(T).Name + "\\" + d + ".vault").Close();
                File.WriteAllText(VAULT_PATH + "\\" + typeof(T).Name + "\\" + d + ".vault", data);

                //increment the index

                index.Add(d, id.ToString());


                //unlock the index file

                saveAndUnlockIndex(typeof(T).Name, index);

                return true;

            }
            catch (Exception ex)
            {

                
                Tracer.Trace(Tracer.severity.ERROR, "Error while inserting Transactions \n\n" + ex.Message+"\n"+ex.InnerException);

                //unlock the index file

                saveAndUnlockIndex(typeof(T).Name, index);
                
            
            }


            return false;
        }

When inserting an object into the vault, the client must first lock the index file by calling getIndexAndLock.

The lock is created by inserting a temporary file named idx_lock.

If the index is locked, the client will loop for a moment then will return an error with a failure to insert the object.

Since there is no reason for the application to monopolize access to the index file, this is acceptable behavior. Of course, a ‘real” implementation may use multithreading and mutexes for instance.

The function cipherObject will serialize the object and cipher it and finally convert it to base64 data.

Finally, once the insertion has been checked, the client will call the saveAndUnlockIndex function to release the index file.

We still need to develop the read and update operations.

To read an object in the vault we need to know its datavault serial. We can then convert the serial into a record number from the IDX file and get the corresponding object.

The following code returns a list of objects from a record number.

/// <summary>
        /// 
        /// Returns a list of objects as stored in the vault
        /// </summary>
        /// <param name="start"></param>
        /// <param name="end"></param>
        /// <returns></returns>
        public static List<T> getObject<T>(int start, int end, DataVaultKey key) where T : dataVaultStorage 
        {

           

            string table = typeof(T).Name;


            if (start > end)
                return null;

            //check password

            if (!checkCredentials(key.user, key.password))
            {
                Tracer.Trace(Tracer.severity.WARNING, "error using credentials " + key.user + "/" + key.password);
                return null;
            }


            List<T> list = new List<T>();
            T object_record;
            string record = "";
            string guid = "";
             Dictionary<int, string> index = null;

            // index = getIndex<T>();

             index = getIndex(table);

            //@start_contract
             Contract.Assert(index != null);
            //@end contract

            for (int num = start; num < end + 1; num++)
            {

                try
                {


                    //  guid = index[num];

                    record = File.ReadAllText(VAULT_PATH + "\\" + table + "\\" + num + ".vault");
                    object_record = decipherRecord<T>(record, key);
                    list.Add(object_record);

                }
                catch (Exception ex)
                {

                    Tracer.Trace(Tracer.severity.CRITICAL, "data vault corrupted, cannot read record " + num + "(" + guid + ")" + "\n\n\n" + ex.Message);
                    object_record = default (T);
                    object_record.info = "error reading the data";
                    list.Add(object_record);
                }
            }

            return list;
        }

And the following code returns an object from its vault ID:

/// <summary>
        /// 
        /// get an object from the vault using the vault id
        /// </summary>
        /// <typeparam name="T">type of the object to fetch</typeparam>
        /// <param name="vaultid">vault id of the object to fetch</param>
        /// <param name="key">key for vault access</param>
        /// <returns>object to be fetched</returns>
        public static T getObjectFromVaultID<T>(string vaultid, DataVaultKey key) where T : dataVaultStorage
        {

            int n = getIndexFromVaultID<T>(vaultid);

            if (n == -1)
                throw new Exception("Object not found Vaultid=" + vaultid);

           return  getObject<T>(n, key);


    }

The update operation is simple. It consists of replacing the object with an updated copy.
The delete operation consists simply of marking the object as deleted. Since all objects to be stored implements the dataVaultStorage class, they have a field for such deletion information.

Testing the Data Vault

Testing the Data Vault

We develop a small program that will call the vault library. We will create several objects, update them and delete some of them.

DataVault.insertObject<Employee>(Anna, key);

            //modify anna

            Anna.Age = 25;

            DataVault.updateTransaction<Employee>(Anna, key);

            //check the modification have been stored

           Employee Anna2= DataVault.getObjectFromVaultID<Employee>(""+Anna.vault_id, key);

           
                Console.Out.WriteLine("employee name="+Anna2.Name+" age="+Anna2.Age);

            DataVault.deleteObject<Employee>(""+Anna.vault_id,key);

            Employee Anna3 = DataVault.getObjectFromVaultID<Employee>("" + Anna.vault_id, key);

            Console.Out.WriteLine("employee name=" + Anna3.Name + " deleted=" + Anna3.deleted);

We check that the object was created and updated accordingly.

employee name=Anna age=25
employee name=Anna deleted=True

Here we delegate the fact that the record is deleted to the developer and let him decide what to do with that information. Also as mentioned previously we let the developer decide how to handle the objects, for example, organizing them in tables, etc…

Conclusion: We were able to create very fast a small but efficient object vault that stores any C# object. Such a technique could be used when there is no reason to use complex databases which can be costly in time with malfunctioning drivers. Sometimes it’s a good idea for a developer to write his own tools rather than counting on third-party software vendors.

Acodez is a leading website design and web development company in India. We offer all kinds of web design and web development services to our clients using the latest technologies. We are also a top digital marketing agency providing SEO, SMM, SEM, Inbound marketing services, etc at affordable prices. For further information, please contact us.

Looking for a good team
for your next project?

Contact us and we'll give you a preliminary free consultation
on the web & mobile strategy that'd suit your needs best.

Contact Us Now!
Jamsheer K

Jamsheer K

Jamsheer K, is the Tech Lead at Acodez. With his rich and hands-on experience in various technologies, his writing normally comes from his research and experience in mobile & web application development niche.

Get a free quote!

Brief us your requirements & let's connect

1 Comment

  1. Trupti

    Hello,

    Am really amazed by this article,
    am very glad that I saw this article.
    Thank you!! Good job on posting this article.

Leave a Comment

Your email address will not be published. Required fields are marked *