Tuesday, August 13, 2013

Atomic write in c++ using one of gnu extensions, boost iostreams, or qt.

Practical snippets for loosely atomic writes on linux
Important, even more so on allocate on flush file systems such as xfs and ext4.
You can learn more here: https://www.flamingspork.com/talks/
https://www.kernel.org/doc/Documentation/filesystems/ext4.txt

These snippets are not perfect, if writing cross platform applications - marco guards

The basic routine is:
1. Write data as file.new
2. write data
3. flush user space buffers
4. sync from kernal space to hardware
5. hope the hardware does what you want
6. rename the file, rename is basically atomic (in posix land)

the close to c++ way:
    assert(__GNUG__); //gnu c++ specific code
    //setup ofstream with an fsync friendly filebuf for those allocate-on-flush
    // filesystems (xfs,ext4,etc)
    const std::string filename_tmp(filename+".new");
    FILE *fp;
    fp=fopen(filename_tmp.c_str(), "w");
    //to fsync when using ofstream
    __gnu_cxx::stdio_filebuf<char>
        linux_fb(fp, std::ofstream::out | std::ofstream::trunc);

    std::ofstream os;
    os.std::ios::rdbuf(&linux_fb); //associate ofstream with gnu basic_filebuf
    //the ofstream methods will not behave as expected from here out.

    assert(linux_fb.is_open());
    if (linux_fb.is_open()==false){
        //fail or whatnot
    }
//... use the ostream to write data
    if(sysconf(_POSIX_FSYNC)) //or use as a predefined macro
    {
        assert(linux_fb.fd());
        if(linux_fb.fd())
        {
            // close (per section 27.8.1.3 c++ std) will flush characters
            // and disassociate linux_fb from ofstream
            os.close();
            if(os.good())
            {
                fsync(linux_fb.fd()); // sync to hardware
            }
        }
    }
    else
    {
        os.close();
    }
    linux_fb.close();


    int fail_status;
    fail_status = rename(filename_tmp.c_str(), filename.c_str());
    if(0 != fail_status)
    {
       //fail
    }

boost iostreams:

#include <boost/iostreams/device/file.hpp>
#include <boost/iostreams/device/file_descriptor.hpp>
#include <boost/iostreams/stream.hpp>
namespace io = boost::iostreams;

std::string fileName_tmp = fileName + ".new";
io::file_descriptor_sink saveFile( fileName_tmp, std::ios_base::out );
io::stream_buffer<io::file_descriptor_sink> stream(saveFile);

 std::ostream toFile(&stream);
 toFile << "Hello, my tuppentup friend";

 if(sysconf(_POSIX_FSYNC))
 {
     toFile.flush();
     if(toFile.good()) //if flush did not fail
     {
         fsync(saveFile.handle());
     }
 }
 stream.close(); //close file
 rename(fileName_tmp.c_str(), fileName.c_str());

Qt QFile:
std::string fileName_tmp = fileName + ".new";
QFile outFile(fileName_tmp.c_str());

if ( outFile.open(QIODevice::Truncate | QIODevice::WriteOnly) )
{
   QTextStream outStream( &outFile );
   outStream << "hello, my tuppentup friend";

   if(sysconf(_POSIX_FSYNC))
   {
       outStream.flush();
       if(outFile.flush())
       {
           fsync(outFile.handle());
       }
   }
   outFile.close();
   // See ksavefile.cpp from KDE project for more about QT rename shortcomings
   outFile.rename(fileName.c_str()); // This is not atomic like a POSIX rename
}


One could also macro guard with #if _POSIX_FSYNC

Tuesday, July 31, 2012

waiting for Xvfb to initialize

I had a problem today where trying to load things into an Xvfb was not working because the xvfb was not yet initialized. The problem script is as follows:

#Start the Xvfb on DISPLAY :1, 
`dirname $0`/Xvfb :1 -screen 0 1280x1024x24 -ac -br -fbdir /dev/shm & xvfbpid=$!
DISPLAY=:1 xli -onroot -fillscreen `dirname $0`/images/background.png

The only solution I see on the web is to insert a sleep statement. I don't much like that. The solution I came up with is to wait for the display to be ready to use via xdpyinfo. The working script: 


#Start the Xvfb on DISPLAY :1, 
`dirname $0`/Xvfb :1 -screen 0 1280x1024x24 -ac -br -fbdir /dev/shm & xvfbpid=$!
#Wait for Xvfb to initialize
ACTIVE=9999
while [ $ACTIVE -ne 0 ] ; do
        xdpyinfo -display :1 &> /dev/null
        ACTIVE=$?
done
DISPLAY=:1 xli -onroot -fillscreen `dirname $0`/images/background.png