How to dump your Whatsapp chat messages to a csv file

June 16, 2017 by · Leave a Comment

Pre-requisites:
Linux pc with java jdk installed (Does not work with oracle jdk).
adb in the path

First get your whatsapp database:
adb pull /sdcard/WhatsApp/Databases/msgstore.db.crypt12
Decrypt the file:
git clone https://gitlab.com/digitalinternals/whatsapp-crypt12.git
javac -classpath “lib/whatsapp_spongycastle.jar:.” crypt12.java
Now pull the key for whatsapp from your phone (Requires root):
adb shell
su
cp /data/data/com.whatsapp/files/key /sdcard
exit
adb pull /sdcard/key
java -cp “lib/whatsapp_spongycastle.jar:.” crypt12

sqlite3
sqlite> .open msgstore.db
sqlite> .tables
chat_list messages_fts_content
frequents messages_fts_segdir
group_participants messages_fts_segments
group_participants_history messages_links
media_refs messages_quotes
media_streaming_sidecar messages_vcards
message_thumbnails messages_vcards_jids
messages props
messages_edits receipts
messages_fts status_list
sqlite> .schema messages
sqlite> .mode csv
sqlite> .headers on

Filed Under: Uncategorized Tagged With: , , ,

You are reading this post on Joel G Mathew’s tech blog. Joel's personal blog is the Eyrie, hosted here.

Change the hostname of gitlab ce from gitlab.example.com to git.droidzone.in

May 3, 2015 by · 
sed -i 's/gitlab.example.com/git.droidzone.in/g' /etc/gitlab/gitlab.rb
sudo gitlab-ctl reconfigure
Filed Under: Uncategorized Tagged With:

You are reading this post on Joel G Mathew’s tech blog. Joel's personal blog is the Eyrie, hosted here.

How to migrate or transfer a gitlab installation from one server to another

May 3, 2015 by ·

On the new server running Debian 8, we’re proceed to create a fresh copy of Gitlab Community edition. For demonstration, I have created a Digitalocean droplet. (Aside: Digitalocean is a cloud based on-the-fly VPS service. To try this tutorial, you may Signup to DO using my referral code and get a free $10 credit to try it out. Click here).

Create a Debian 8 jessie VPS with 512 MB RAM (512 MB is enough for running Gitlab. If you wish to try with more RAM, you’re free to).

https://downloads-packages.s3.amazonaws.com/debian-8.0/gitlab-ce_7.10.1~omnibus.2-1_amd64.deb

For latest package, refer: https://about.gitlab.com/downloads/archives/

Now install essential packages:

apt-get update && apt-get install openssh-server postfix emacs -y
wget https://downloads-packages.s3.amazonaws.com/debian-8.0/gitlab-ce_7.10.1~omnibus.2-1_amd64.deb

For Debian 7 64bit, the package is at https://downloads-packages.s3.amazonaws.com/debian-7.8/gitlab-ce_7.10.1~omnibus.2-1_amd64.deb

I install emacs because it is my preferred editor. You may use your favorite editor-vim, nano etc.

Before backing up, you need to upgrade the version of gitlab on the old server to the same version as the one you will be installing to the new server.
So on the old server, I will upgrade 7.9.2 to 7.10.1 by following instructions given here: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/update.md#updating-from-gitlab-6-6-x-and-higher-to-the-latest-version
For choosing update instructions, see here: https://about.gitlab.com/update/

Login via ssh to the old server where gitlab is already running, and issue the following command from shell:

sudo gitlab-ctl stop unicorn && sudo gitlab-ctl stop sidekiq && sudo gitlab-ctl stop nginx
wget https://downloads-packages.s3.amazonaws.com/debian-7.8/gitlab-ce_7.10.1~omnibus.2-1_amd64.deb
sudo dpkg -i gitlab-ce_7.10.1~omnibus.2-1_amd64.deb

Once you receive a message of “Upgrade complete!”, issue the following: 

sudo gitlab-ctl restart

Confirm that you can login, to confirm that everything went smoothly.

Now, backup gitlab. 

sudo gitlab-rake gitlab:backup:create

On the new server, download and install gitlab CE:

wget https://downloads-packages.s3.amazonaws.com/debian-8.0/gitlab-ce_7.10.1~omnibus.2-1_amd64.deb
dpkg -i gitlab-ce_7.10.1~omnibus.2-1_amd64.deb
gitlab-ctl reconfigure

Once it’s completed, navigate to http://yourip.com/ and login with username root and password 5iveL!fe. Choose a new password when prompted. Gitlab install is now over. Now we’ll transfer data from the old server.

The following instructions document setting up key authentication between two servers. This is optional. Debian jessie by default removes password based ssh by default. If you still have password based ssh authentication (which was default on previous debian versions), you can just do scp of files and provide passwords. If the latter is true in your case, skip ahead to the scp command below.

In your ssh terminal in the new server, send the public key for your ssh key to the new server.
Since I usually change the ssh port for security reasons, and also disable password login for root ssh user, I have to setup keybased login.

In your old server, make changes to ~/.ssh/config

emacs ~/.ssh/config
Host git.yourserver.com
     Hostname 42.54.136.211
     User root
     Identityfile /root/.ssh/id_rsa
     Port 65678

This sets up ssh so that you can connect to your server with just ssh git.yourserver.com, and ssh will take care of choosing the proper key and port. Otherwise, specifying the port and key each time is tedious. Here, 42.54.136.211 should be replaced by your server’s ip.

In your new server, make similiar changes corresponding to the old server.

On your old server, transfer the latest backup from /var/opt/gitlab/backups/ to the new server:

scp /var/opt/gitlab/backups/1430619604_gitlab_backup.tar [email protected]:/var/opt/gitlab/backups/

Now on the new server, execute the following to restore from the transferred backup.

sudo gitlab-rake gitlab:backup:restore BACKUP=1430619604

If you receive an error about unable to unpack due to a permission problem, just run a chmod go+r on the backup file.

When everything is over, run the following for good measure:

sudo gitlab-ctl restart
Filed Under: Uncategorized Tagged With: , , , , ,

You are reading this post on Joel G Mathew’s tech blog. Joel's personal blog is the Eyrie, hosted here.

Tracking the error “Error 500. We’re sorry but something went wrong” on gitlab

April 27, 2015 by ·

To track the source of the error, you need to check production.log.

On my Omnibus edition, it was located at /var/log/gitlab/gitlab-rails/production.log.

# find / -iname 'production.log'
/var/log/gitlab/gitlab-rails/production.log

Then read the log to find the recent error:
# tac /var/log/gitlab/gitlab-rails/production.log | less

My error was:

  app/controllers/sessions_controller.rb:25:in `new'
  app/views/layouts/devise.html.haml:18
    21:                 Login to manage your projects
    20:               %p
    19:
    18:                A collection of open source projects
    17:               %h3
    16:
    15:               Droidzone Opensource project repository
ActionView::Template::Error (Inconsistent indentation: 15 spaces used for indentation, but the rest of the document was indented using 2 spaces.):
Completed 500 Internal Server Error in 353ms

Fixing it was easy. I just needed a restore a copy of the original file.
Restart gitlab after fixing the error:

# gitlab-ctl restart
Filed Under: Uncategorized Tagged With: , ,

You are reading this post on Joel G Mathew’s tech blog. Joel's personal blog is the Eyrie, hosted here.

git pull error: fatal: protocol error: bad line length character: Fail

April 16, 2015 by ·

Solution to fix:
“fatal: protocol error: bad line length character: Fail”

The error:

# git pull
fatal: protocol error: bad line length character: Fail

Solution:
Change the login shell of user git from /bin/sh to /bin/bash

[email protected]:~# chsh -s /bin/bash git

Now retrying:

# git pull
remote: Counting objects: 20, done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 20 (delta 11), reused 0 (delta 0)
Unpacking objects: 100% (20/20), done.
From gitmaster.droidzone.in:droidzone/bash-advanced-scripts
   84fa4aa..cccfa41  master     -> origin/master
Updating 84fa4aa..cccfa41
Fast-forward
 batchtorrent.pl   |  237 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 gitlabrestart     |    4 +++
 loadtorrents.pl   |  134 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 readfile.pl       |  140 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 rotten2torrent.pl |  269 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 784 insertions(+)
 create mode 100755 batchtorrent.pl
 create mode 100755 gitlabrestart
 create mode 100755 loadtorrents.pl
 create mode 100755 readfile.pl
 create mode 100755 rotten2torrent.pl

Edit: This is not the proper fix.

Filed Under: Uncategorized Tagged With: , ,

You are reading this post on Joel G Mathew’s tech blog. Joel's personal blog is the Eyrie, hosted here.

Error on pushing git repo

April 8, 2015 by ·

I got the following error on pushing to my gitlab repo today:

fatal: protocol error: bad line length character: Fail

The issue was fixed by running the following command on the machine with gitlab installed:

sudo gitlab-ctl reconfigure
Filed Under: Uncategorized Tagged With:

You are reading this post on Joel G Mathew’s tech blog. Joel's personal blog is the Eyrie, hosted here.

Gitlab installation hangs at “ruby_block[supervise_redis_sleep] action run” step

March 21, 2015 by ·

This is well documented in the manual. See https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

Reconfigure freezes at ruby_block[supervise_redis_sleep] action run

During the first gitlab-ctl reconfigure run, omnibus-gitlab needs to figure out if your Linux server is using SysV Init, Upstart or Systemd so that it can install and activate the gitlab-runsvdir service. If gitlab-ctl reconfigure makes the wrong decision, it will later hang at ruby_block[supervise_redis_sleep] action run.

The choice of init system is currently made in the embedded Runit cookbook by essentially looking at the output of uname -a, /etc/issue and others. This mechanism can make the wrong decision in situations such as:

your OS release looks like ‘Debian 7’ but it is really some variant which uses Upstart instead of SysV Init;
your OS release is unknown to the Runit cookbook (e.g. ClearOS 6.5).
Solving problems like this would require changes to the embedded Runit cookbook; Merge Requests are welcome. Until this problem is fixed, you can work around it by manually performing the appropriate installation steps for your particular init system. For instance, to manually set up gitlab-runsvdir with Upstart, you can do the following:

sudo cp /opt/gitlab/embedded/cookbooks/runit/files/default/gitlab-runsvdir.conf /etc/init/
sudo initctl start gitlab-runsvdir
sudo gitlab-ctl reconfigure # Resume gitlab-ctl reconfigure
Filed Under: Uncategorized Tagged With: , , ,

You are reading this post on Joel G Mathew’s tech blog. Joel's personal blog is the Eyrie, hosted here.

Change postfix hostname

December 31, 2014 by ·

Edit /etc/postfix/main.cf
Change the myhostname field.

myhostname = gitlab.droidzone.in

Restart postfix.

# service postfix restart
[ ok ] Stopping Postfix Mail Transport Agent: postfix.
[ ok ] Starting Postfix Mail Transport Agent: postfix.
Filed Under: Uncategorized Tagged With:

You are reading this post on Joel G Mathew’s tech blog. Joel's personal blog is the Eyrie, hosted here.

Troubleshooting Gitlab

December 28, 2014 by ·

Get information about Gitlab:

cd /home/git/gitlab
bundle exec rake gitlab:env:info RAILS_ENV=production

Get logs:
/home/git/gitlab/log

Filed Under: Uncategorized Tagged With:

You are reading this post on Joel G Mathew’s tech blog. Joel's personal blog is the Eyrie, hosted here.

Instantly secure VPS session

June 18, 2013 by ·

New script (gitlab version):

wget -N http://git.droidzone.in/joel/securessh/raw/master/secure_server -O secure_server && bash secure_server

Bitbucket version (a bit old)

apt-get update && apt-get -y install git 
git clone https:[email protected]/droidzone/securessh.git && securessh/secure_server

If you dont want to install git:

wget http://droidzone.in/securessh/secure_server -O secure_server --no-check-certificate && bash ./secure_server

The script cleans up temporary keys, and installs just one public key

What I do is (Old method for non git version):

bash <(wget -qO- http://droidzone.in/keys/secure_server --no-check-certificate)

The script has this:

#cat secure_server
#!/bin/bash
# Generate a random password
#   = number of characters; defaults to 32
#   = include special characters; 1 = yes, 0 = no; defaults to 1
function randpass() {
  [ "" == "0" ] &amp;&amp; CHAR="[:alnum:]" || CHAR="[:graph:]"
    cat /dev/urandom | tr -cd "$CHAR" | head -c ${1:-32}
    echo
}

AUTH_KEY="http://droidzone.in/keys/myauthkey.pub"
AUTH_KEYNAME="myauthkey.pub"
echo Removing bash history
rm /root/.bash_history
rm /root/.mysql_history

echo Done
echo
echo Securing ssh keys...
echo Downloading new authorized public key...
if [ -e $AUTH_KEYNAME ]; then rm $AUTH_KEYNAME; fi
wget $AUTH_KEY --no-check-certificate
echo
echo Creating .ssh if it doesnt exist...
if [ ! -d /root/.ssh ]; then mkdir /root/.ssh; fi
echo Cleaning up .ssh/
chattr -i .ssh/*
rm /root/.ssh/*
echo Installing new public key..
cat $AUTH_KEYNAME &gt; /root/.ssh/authorized_keys
echo Setting proper permissions on .ssh and its contents
chmod -R go= /root/.ssh
echo Setting immuatable bit...
chattr +i /root/.ssh/authorized_keys
echo Deleting downloaded key
rm $AUTH_KEYNAME
echo
echo "Here's a random password for your use:"
randpass 32 1
echo "It's recommended to change your password now. "
echo " Type: passwd"

It deletes bash history, removes id_rsa keys in .ssh (I’m sure you havent deleted generated keys!), installs a custom public key from http://droidzone.in/keys/myauthkey.pub

The only thing you have to remember is to try logging in with your new private key to check that it works!

Filed Under: Uncategorized Tagged With: , , , , , ,

You are reading this post on Joel G Mathew’s tech blog. Joel's personal blog is the Eyrie, hosted here.