9.0

Edit

Corruption: truncated record at end of file

Below is a sample line in console.log of Riak:

2018-01-11 14:21:36.712 [error] <0.1049.0>@riak_kv_vnode:init:512 Failed to start riak_cs_kv_multi_backend backend for index 502391187832497878132516661246222288006726811648 error: [{riak_kv_eleveldb_backend,{db_open,"Corruption: truncated record at end of file"}}]

Riak is not starting.

Statement

Riak has a corrupted leveldb database, which can happen after disk space issues and restart.

Solution

  1. Stop Riak.

    service s3emu stop

  2. Back up Riak.

    tar -cvzf /backups/riak-backup`date +%Y%m%d-%H%M`.tar.gz /opt/workfusion/apps/s3emu

  3. Find corrupted vnodes in console.log. In our example, it is 502391187832497878132516661246222288006726811648

    2018-01-11 14:21:36.712 error <0.1049.0>@riak_kv_vnode:init:512 Failed to start riak_cs_kv_multi_backend backend for index
502391187832497878132516661246222288006726811648 error:
truncated record at end of file"}}

  4. Repair all corrupted vnodes in leveldb. Make sure you set correct DataRoot and VNodeList.

    # `riak ertspath`/erl 
> application:set_env(eleveldb, data_root, "").
> Options = [].
> DataRoot = "/opt/workfusion/apps/s3emu/riak/leveldb".
> VNodeList = ["502391187832497878132516661246222288006726811648"].
> RepairPath = fun(DataRoot, VNodeNumber) -> Path = lists:flatten(DataRoot ++ "/" ++ VNodeNumber), io:format("Repairing ~s.~n",[Path]), Path end.
> [eleveldb:repair(RepairPath(DataRoot, VNodeNumber), Options) || VNodeNumber <- VNodeList].


Wait untill vnodes are repaired, then press Crtl+C -> a

  5. Start Riak.

    service s3emu start

  6. Validate that Riak is started and buckets are accessible.

    /opt/workfusion/lib/s3cmd/s3cmd -c /opt/workfusion/apps/s3emu/etc/s3cfgfasttrack ls
← Previous