Well established industrial IT solutions often do not suit the needs of researchers. Industrial technologies for data management and processing, perfectly tailored to serve businesses or banks, often appear weak and clumsy in scientific applications. Thus, in genomics, it is typical that a single analytical job requires retrieving and storing back hundreds of millions of objects like short DNA sequences. With such volumes, overhead of transaction processing inherent for industrial SQL DBMSs becomes unbearable.

SciDM Group provides an alternative - SciDM Database Management System, a zero-maintenance object-relational NoSQL database engine optimized for large aggregate transactions. On typical bioinformatics tasks like sequence assembly and clustering, SciDM Data Manager operates thousands of times faster than conventional database engines. It also provides a rich and flexible data model, native mapping to an array of programming languages, and very efficient over-the-network access. For full details, please see SciDM Whitepaper[PDF].

Scientific Data Manager (SciDM) is a DBMS designed for applications that require:

• to operate quickly with heavy data sets over a network -
• on standard inexpensive hardware, free of maintenance -
• through light and easy-to-use programming interface -
• using multiple platforms and various programming languages.

SciDM provides:

- Unmatched performance:

retrieving 1,000,000 data objects in 0.9 seconds

updating 1,000,000 data objects in 2.1 seconds

creating 1,000,000 data objects in 37 seconds

deleting 1,000,000 data objects in 21.5 seconds

processing 1,000,000 select queries in 5.2 seconds

handling 10,000 client sessions simultaneously

- over a network¹, running on a standard hardware², with 540Gb of data pre-loaded³.

- Strong reliability: All features are tested in real-world applications that intensively operate with huge volumes of data. These applications include genomic data management, literature databases, document management and bug tracking.

- Truly unlimited storage – limited by hardware only. SciDM-managed database of 12.5 Terabytes exists. The storage is theoretically limited to 2⁴⁸ objects; each object can contain up to 2 billions of data attributes; each data attribute can contain up to 2⁶³ bytes.

- Rich data management capabilities at unprecedented speed. It uses an object-relational model for data representation and provides methods for storing, retrieving, and deleting data objects, for sequential and indexed access and for dynamic data structure discovery. Along with traditional indexing by entire attribute contents, SciDM provides the ‘context’-style indices by the individual words in stored texts.

- Structured framework for data processing formalized in terms of a particular application field. This makes SciDM highly suitable for research and prototyping. This also simplifies the development by removing the traditional data translation layer between application and DBMS, and by bringing the structured data directly into the processing modules.

- Interoperability over various platforms, operating systems and programming languages.

- Security model with object–level protection. The access rights are controlled individually for every object.

- Among other features of SciDM there are: flexible object structure, allowing dynamic addition of new attributes to existing objects; integrated set management, allowing both persistence and server-side operations for arbitrary objects sets; data integrity support through locking and object subordination.

We are presently starting a company, SciDM Co., for commercialization the SciDM database engine. Please see scidm.com for details.

Footnotes:

EMBEDB is a high-performance data storage and access library, written in C++. It serves as a back end for the SciDM (Scientific Data Manager) DBMS.

EMBEDB features:

• Tightly optimized performance
• Native C++ implementation
• File system virtualization layer for cached paging
• Time- and space-efficient variable-length records management
• Flexible Btrees with compressable key/values storage (through range encoding), allowing for amortized constant time access to objects by Id (and in a class of other practical cases)

EMBEDB is an open-source progect, available for download on GitHub.

QSimScan (Quick SIMilarity SCANner) is a technology for very fast biosequence similarity search. It is a foundation for two software tools:

• NSimScan, for searching nucleotide sequence similarities and
• PSimScan, for searching protein sequence similarities.

These tools perform 5 to 100 times faster than standard NCBI BLAST, depending on chosen parameters. Their sensitivity and selectivity are comparable to NCBI BLAST at the slowest settings, and decrease with the increase of speed, however, they are maintained at the levels reasonable for most tasks.

QSimScan has the greatest advantage when used on large collections of query sequences. Comparing entire proteome of E.coli (4,132 proteins) to the NCBI non-redundant protein database of 6,441,864 records takes 1.5 hours on moderately powerful laptop computer (over 76 hours with NCBI BLAST).

Reference implementations for PSimScan and NSimScan are open source, available for download on GitHub.

QSimScan is presently used in both industrial and academic environments for tasks like whole genome annotation, protein clustering and classification, contamination filtering, short reads assembly, EST clustering, and others.

SciDM Group's EST Pipeline is an integrated solution for transcriptomic data processing, capable of analysis of large EST and RNAseq datasets. The pipeline integrates custom high-performance software components for:

• Quality and composition-based trimming
• Repeat detection and masking, bundled with repeat database construction and management
• Fast all-to-all comparison and similarity matrix computing (NSimScan-based)
• Very efficient clustering and scaffolding with configurable link selection rule sets
• Elaborate Chimera detection, based on local cluster topology and coverage composition
• Contig assembly, splice variants resolution and processing path reconstruction
• Virtual expression profiles construction
• Similarity-based annotation of assembled transcripts

The pipeline is implemented as a set of application servers capable of running on separate network nodes and communicating through CORBA layer, and controlled over a network by a remote dispatcher. The pipeline uses SCIDM as a data storage back-end.

Denis Kaznadzey

Computational Biologist and Software Engineer

Denis graduated from Moscow State University in 1989 with major in Biochemistry. In 1991, together with Vlad Novichkov and others, he founded Query Logic, Inc. - a company that brought to market the world's first hardware accelerated DNA sequence analysis suite - ImaGENE. Over the course of his career, Denis led the development of large-scale bioinformatics systems for genome annotation, metabolic modeling, transcriptome assembly, gene expression analysis, and developed algorithms for solving various problems in computational biology. Denis sees "reverse-engineering of life" as his major mission and challenge.

Victor Joukov

Software Architect and Algorithm Designer

Physicist by education and software guru by talent, Victor started his career at ParaGraph, a high-tech software company in Russia, specializing in human-computer interfaces. Victor entered the world of Life Sciences in 1988, when he joined Quark Biotech's Bioinformatics Group as a Lead Software Architect. Victor has developed and delivered a variety of highly usable products and technologies for genome annotation, gene function prediction, analysis of biological networks, comparative genomics, metabolic engineering and other fields. Presently, as Senior Software Engineer at the National Institutes of Health, Victor is responsible for the development of the NCBI Sequence Viewer.

Vladimir Novichkov

Computer Scientist, Electrical Engineer.

Vladimir has studied Physics in Moscow State University. In 1991 Vladimir, together with Denis Kaznadzey and others, co-founded Query Logic, Inc., that pioneered the design of a specialized hardware co-processor for DNA sequence comparison. While directing the microarray facility at UIC / Quark Biotech, he developed tools for microarray image analysis and information extraction. He also designed algorithms for ultra-fast DNA sequence comparison, which later inspired the development of, and became a foundation for QSimScan technology. In later years Vladimir worked on hardware architecture and algorithms for networking, signal processing and wireless channel coding, and co-authored several US patents and a conference paper. Among Vlad's current research interests are special purpose processor microarchitectures, graph and supervised learning algorithms.