Monday, December 21, 2020

Fighting Mongodb after an unexpected shutdown - WiredTiger.turtle permission issues

 On a sandbox server that we manage, a Mongodb based application that we manage was affected by an unexpected shutdown. While I have these notes in another location, they are, unfortunately, behind a login and therefore not easy for others on the internet to access. I'm dumping these logs here for the benefit of my future self.

In summary:

The WiredTiger.turtle file was owned by root when it should have been owned by the user associated with MongoDB

I'm looking in db/logs/db.log and seeing the following

2020-12-21T14:23:52.569+0000 I CONTROL  [initandlisten] build environment:
2020-12-21T14:23:52.569+0000 I CONTROL  [initandlisten]     distarch: x86_64
2020-12-21T14:23:52.569+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2020-12-21T14:23:52.569+0000 I CONTROL  [initandlisten] options: { net: { port: 6001 }, processManagement: { fork: true }, storage: { dbPath: "db/data" }, systemLog: { destination: "file", path: "db/logs/db.log" } }
2020-12-21T14:23:52.593+0000 I -        [initandlisten] Detected data files in /home/godata/db/data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-12-21T14:23:52.594+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2020-12-21T14:23:52.609+0000 E STORAGE  [initandlisten] WiredTiger (13) [1608560632:609444][21617:0x7f28ae23acc0], file:WiredTiger.wt, connection: /home/godata/db/data/WiredTiger.turtle: handle-open: open: Permission denied
2020-12-21T14:23:52.610+0000 I -        [initandlisten] Assertion: 28595:13: Permission denied
2020-12-21T14:23:52.610+0000 I STORAGE  [initandlisten] exception in initAndListen: 28595 13: Permission denied, terminating
2020-12-21T14:23:52.610+0000 I CONTROL  [initandlisten] dbexit:  rc: 10

I'm most concerned about the permission denied

2020-12-21T14:23:52.610+0000 I STORAGE  [initandlisten] exception in initAndListen: 28595 13: Permission denied, terminating

All files are owned by the `godata` user (the user responsible for starting all processes, so I'm not sure why I'm getting permission denied.

I tried repairing the db

platforms/linux/x64/default/mongodb/bin/mongod --dbpath db/data --repair

The output still gives me a permission denied

2020-12-21T14:39:33.590+0000 I CONTROL  [initandlisten] MongoDB starting : pid=4224 port=27017 dbpath=db/data 64-bit host=ubuntu-s-2vcpu-4gb-nyc1-01
2020-12-21T14:39:33.591+0000 I CONTROL  [initandlisten] db version v3.2.21
2020-12-21T14:39:33.591+0000 I CONTROL  [initandlisten] git version: 1ab1010737145ba3761318508ff65ba74dfe8155
2020-12-21T14:39:33.591+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2020-12-21T14:39:33.591+0000 I CONTROL  [initandlisten] modules: none
2020-12-21T14:39:33.592+0000 I CONTROL  [initandlisten] build environment:
2020-12-21T14:39:33.592+0000 I CONTROL  [initandlisten]     distarch: x86_64
2020-12-21T14:39:33.592+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2020-12-21T14:39:33.592+0000 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "db/data" } }
2020-12-21T14:39:33.619+0000 I -        [initandlisten] Detected data files in db/data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-12-21T14:39:33.619+0000 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2020-12-21T14:39:33.619+0000 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=1G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2020-12-21T14:39:33.644+0000 E STORAGE  [initandlisten] WiredTiger (13) [1608561573:644475][4224:0x7f28b011ccc0], file:WiredTiger.wt, connection: db/data/WiredTiger.turtle: handle-open: open: Permission denied
2020-12-21T14:39:33.645+0000 I -        [initandlisten] Assertion: 28718:13: Permission denied
2020-12-21T14:39:33.646+0000 I STORAGE  [initandlisten] exception in initAndListen: 28718 13: Permission denied, terminating
2020-12-21T14:39:33.646+0000 I CONTROL  [initandlisten] dbexit:  rc: 100
I've found a similar issue documented here:

ls -l db/data/WiredTiger.turtle

-rw-r--r-- 1 root root 1011 Dec 21 10:48 db/data/WiredTiger.turtle

As the root user I ran the following command (note.. my setup uses a user user called 'godata', this will vary for others)

chown -R godata:godata /home/godata/db/

I was then able to succesfully launch the standard Go.Data scripts.

Thursday, November 1, 2018

NCB Digital Symposium 2018 - Terry Ann Segree - Digital Transformation and Small Businesses

Small businesses are already agile [by default]. They are working in small groups, need to move quickly etc…

We are now in a digital economy. We use our mobile phones and everyday new innovations are developed. Regardless of size you're not protected from the disruption that is happening.

To achieve digital transformation you need to make the time otherwise you will be out of business.
Digital transformation is enabled by technology but driven by people. This includes founders, employees, customers, suppliers and partners. As a small business you don't have the resources to hire a "Digital Transformation Officer".

Focus on strategy and culture

  • Lead from the front
  • Put strategy before technology
  • Encourage new cultural norms
  • Competence of team
If you don't foster the culture where your employees relate to you what they have learnt from
The customer you are losing some of the opportunities [to learn from customer feedback].
Your customers want it now! They don't want to wait, this is true of employees, they want
To feel a part of the ... (not sure what she said here).

The process of innovation

From Thinking in new boxes:
  1. Doubt everything
  2. Explore
  3. Diverge
  4. Converge
  5. Re-evaluate

BIC pens in a meeting asked the question.. Why not make razors?
Look at your business. When Spirit airlines started they started as a low-cost carrier with now assigned seating.
You need to understand the world you live in, not just your business. Understand your customers and look at how to solve their problems.

(missed the rest of the presentation)

NCB Digital Symposium 2018 - Warren Vernon - National Identification System

National Identification System

Enabler of Digital Economy
Warren Vernon

I had to step out so I didn't get the whole presentation. Employees working with data can get up to 25 years of imprisonment for unauthorized verification or
unauthorized internal access to individuals' personal data.

Applications of NIDS(not his term)


  • Visual, face-to-face identification
  • Information on the face of the card + a single fingerprint stored on the NIC;
  • Match on Card fingerprint automatic identification
  • NIC contains a digital certificate to prove the identity of the card holder
  • Use of the card holder PIN + Biometric to authenticate

Digital Signature
  • Chip contains a second certificate 
  • Signature can be applied to electronic document

Banking as a service

In the future you'll be able to do onboarding online, sign contracts online, possibly use biometric authentication at Automated Banking Machines.

Opportunities for Regulators and Stakeholders
  • Fast-track public consultation to establish guidelines for applicants
  • Take swift actions from consultation to application
  • Issue/take up the first local Virtual Bank License
Opportunities for the 1st Virtual Bank and Customers
  • Promote financial inclusion
  • No physical branches
  • Eliminate minimum and low account balance fees
  • Target small and medium sized enterprises (SMEs) and Low income individuals
  • Establish digital banking services for Jamaicans living overseas
  • Reduce risk for digital IDS
Authentication is done based on what a person has or knows; Some confidence; High risk of fraud
Authentication is done based on who a person is (Iris, Fingerprint, Face);
Very high confidence Low risk of fraud.

Opportunities for e-Prescriptions, Feeless Real Estate transactions

A National Accreditation Certificate Authority

(missed details here)

LIfecycle Data Exchange

  • Identity data entered once at NIRA
  • Contact info, Identity and Name Change Sharing
  • Current and Historical Addresses Sharing
  • LIfecycle change data sharing - marriages, deaths etc.

Transform Government Procurement

  • Create fully digital procurement Portal
  • Hide seller information before a match is established
  • Automate the matching of business needs or specifications with vendor products
  • Expand the local e-commerce industry

Reduce corruptions; increase innovation; increase transparency; Public Private Partnership to build it out.

Addresses as a service

  • Introduction of a National Standardized Address Database
  • Public and Private Sector will have access to use standardized addresses for consistency
  • No personal information in the database

Improve access for Persons with Special Needs

  • Proof if government benefits and welfare are reaching disabled persons
  • Create opportunities for the disabled

NCB Digital Symposium 2018 - Belkis Vasquez-McCall - Jump starter for large corporates

Jump starter for large corporates

Building agility in digital transformations
Belkis Vasquez-McCall

What does it take for large organisations to be a catalyst across the industry. The gap between when
you have an idea and when you get it to the customers has been drastically reduced. Innovation today
can reach 50 million users faster than ever.

Radio to 38 years to reach 50 million users while smartphones took less than a year to do the same.
Digital is fundamentally changing the way things work along the value chain.

Digital maturity correlates to financial metrics.

[Amazon Go video]

The key elements of the digital ecosystem

It takes a holistic approach. Start by understanding the integrated and disruptive ecosystem. The core
pillars of enterprise-wide digital transformation are: Organisational redesign, digital talent and technology
enablers. Customer-centric journey reimagined.

The mindset needs to be different, even as a developer.
How do make sure that as leaders you're modeling the change you want to see? How do you develop
an agile mindset? Agility is being able to be responsive to change, work with you customers to get
feedback and adjusting based on feedback.

We're moving away from an "organisation as a machine" to "organisation as organic system".

What is your big dream that you're going to bring back to your organisation and really push the thinking?

NCB Digital Transformation Symposium 2018 - Nadeen Matthews-Blair - Creating a Digital Transformation Programme

Creating a Digital Transformation Programme

Nadeen Matthews-Blair

NCB embarked on its digital transformation journey in 2015 with three objectives
  1. Enhance Customer Experiences
  2. Enhance efficiency
  3. Transform business model for long-term viability


  • Disrupt or be disrupted
  • Digital technology is the new equalizer

Disrupt or be disrupted

Examples, blockbuster, kodak, airbnb vs hotels etc….

Digital technology is the new equalizer

Examples, Estonia, Alibaba, Transferwise. It is no longer "owned" by Silicon valley.

At NCB we believe we can leverage, mobile technology, open source technology to meet our objectives.

What is digital transformation?

It is not about a new App, Website, becoming Paperless, ERP or CRM.

We define it as..

the transition from a product centric business model to a customer journey centric platform model, characterised by simplicity, ease, speed, convenience, elight and powered by digital technology, artificial intelligence and partnerships.

A good example is John Deere who moved from a manufacturer to a platform.

How do we become more concerned about the customer journey?

2015-2016 - defined our ambition, launched new digital products, evaluated and refined approach We borrowed from Jeff Bezos, be more customer focused. We launched a few digital products and learned some lessons to help us refine the approach (not the ambition). With that introspection we decided we needed a new approach. It was taking us too long to get to market and we needed to do something different.

2017 - Launched 1st Agile Lab, Accelerated digital transformation, evaluated approach. By the end of 2017 we had 4 agile labs.

2018 - We scaled up and we are now at 15 agile labs.

The agile way of working was formally codified by software developers in 2001 has helped us to get to market faster. For example:

Greater speed to market

  • Credit card alerts 10 weeks
  • Account opening - Deposits were developed in 22 weeks

Enhanced Team collaboration
IT is no longer in a silo, cross functional teams are collaborating on common goals. These are not senior people.

Greater customer centricity and adoption

We constantly test solutions with the customer instead of a big role out and feedback later. Every week or two weeks we do internal demos and every 4 to 6 weeks we test with customers. We launch in increments and test. For example our account opening solutions addresses 40% of our customers. Instead of waiting we're rolling out in increments.

I can't stress enough the empowerment. Our executives are not making the decisions that happen in the lab, instead the teams are making decisions based on defined goals. If you want

Hallmarks of Agile

  1. Collaboration and empowerment
  2. Customer centricity
  3. Test and learn/ Continuous improvement
  4. Lessons Learnt
  • It is not about the technology
  • Tremendous value to be unlocked by bringing together and empowering cross functional teams
  • Focus on end-to-end journeys
  • Partnerships are key
  • Digital and Agile are mindsets
  • Ensure team understands case for change and share successes
  • Continuous learning and improvement is the name of the game - learn it all not know it all

Example.. We've partnered with eGov so we can plug in to their data to pull customer information to save them the time of filling out a form.

Your team must understand the case for change

  1. Talent
  2. ICT
  3. Culture Transformation
  4. Cybersecurity
  5. We went from a total of 20 application developers to needing to recruit 30 in the first year.

We partnered with local IT companies who were better at getting talent. When we looked at the pipeline of talent that we needed there wasn't enough talent. We partnered with the University to help to prepare talent. Not just developers, data scientists and UX professionals.

We had infrastructural issues. If you have ICT based platforms and FLOW goes down that's a problem. It is not a resources issue, it's about leadership and commitment. This requires a huge cultural transformation from "this is how it is done" to how we focus on what the consumer needs and wants and how to get there.

We are proactively putting this in place to protect customers.

Getting started

If it doesn't sound like science fiction it isn't crazy enough!
  • Get buy-in from the CEO 
  • Establish digital transformation ambition 
  • Create your change story that articulates the case for transformation 
  • Start small - prioritise and select a customer journey for re-imagination and select cross-functional team with right mind-sets to address the problem 
  • Scale up quick wins and share success 
  • Continue to test, learn and improve 
Remember the disruption may come from outside your industry. Choose a journey that will impact your customers and business but not something that has too much technical or ecosystem dependencies. This allows you to develop the muscle and then you will have an early success story to use as a case for change within your organisation. Once you have that quick win then you can scale up and address other areas of your roadmap. This is a process of continuous improvement and continuous learning.

Next Steps

  • Scaling the agile way of working 
  • Leveraging advanced analytics and artificial intelligence 
  • Moving from product to platform
We would love to become the Google of the region or the world.

Live Blog NCB Digital Symposium - Dr Lila Rao-Graham Jumpstarting your Digital Transformation

Jumpstarting your Digital Transformation

Dr. Lila Rao-Graham

This such an important initiative. This conversation is a cr

IN our local context are we ready
The CEO spoke to some important components of that.
WE're trying to understand.

What is strategic digital leadership?
What does it entail?

Who is the leader?
Other execs.

Answer… it's about the relationship/partnership between the leaders. The skills needed for the CIO is not just about the technology. That myth needs to go away now. It is important that the CIO acts as a strategists on how to use technology to address business issues.

It is important that the partnership is developed and the CEO pushes the [digital] agenda and support the work of the CIO.

It's NOT about the technology. The technology is a part but it is the integration of the technology with the strategy. How does the technology become a part of the strategy. You have to develop the capabilities needed and invest in your talent. NCB has been very strategic in this, they've come to the university to engage to make sure graduates are coming out with the skills they need.

It requires a change in
  • Strategy
  • Organization & Talent
  • Capabilities
  • Culture

To quote Lego on digital transformation - "No more digital strategy - executing strategy digitally"

It's note an option now for companies it is an imperative. Some of these changes don't require large investments so it can impact small players as well. It requires a change in mindset.

What is digitalization?
It isn't the same for all. It is important to define it.

Size Matters, but…

Remember Blockbuster vs Netflix. It isn't about the big fish, it's about the fast fish. Netflix disrupted the video rental industry and killed Blockbuster.

Adapted a survey administered by MIT Sloan & Deloitte
  • Used a convenience sample
  • 60 responses
  • 1 year period

Most organisations didn't have a clear digital agenda.
Most organisations (53%) considered themselves developing, which I think is optimistic.

You have to be open to this kind of radical change. Leadership is essential, you need to expand the role of mobile and mobility (how are your customers consuming your data and information).

Core technologies:
  • Social
  • Mobile
  • Analytics
  • Cloud

It is an integrated approach:
No longer working silos
  • Strategy
  • Culture
  • Capabilities
  • Org & Talent

My favourite quote:

We are drowning in information, while starving for wisdom. The world henceforth will be run by synthesizers, people able to put together the right information at the right time, think critically about it, and make important choices wisely.
– Edward Osborne Wilson

I want to congratulate NCB. This symposium is a critical success factor.

Live blog NCB Digital Transformation Symposium - Ruel Reid - Up-skilling and Re-skilling for a Digital Society

Up-skilling and Re-skilling for a Digital Society

Ruel Reid

Good morning

I do embrace technology and I'm considered to be a disrupter, just tell me what the problem is and I'll take on all the challenges and [make it happen]

I've been discussing how we in education have to embrace technology.

Our prime minister is passionate about digital transformation. We are in the business of developing our human resources. We need to ensure our system is producing the humans needed for this change.

Humans change slowly (evolutionary)
Technology changes quickly (revolutionary)

We are seeing ICT innovations with IoT, Artificial Intelligence, Virtual Reality and Augmented Reality, Big Data and Blockchain.

The united nations has as sustainable development goal "Quality education".

Putting on my HR hat, I've been advocating a mandatory 30 hours of professional development for every employee to promote lifelong learning for all. This aligns with our strategic plan vision 2030.

The ministry has achieved the following:
  • MoEYI Strategic direction for new and emerging technologies 
  • National ICT Competency Framework for Teachers and Curriculum that will ensure that teachers have the basic skills required to prepare students 
Examples of transformation:

Was a forerunner in implementing a digital school management system at Jamaica College, there was a time when we had to enter grades on broadsheet paper reports, now you can be [anywhere] and upload the data.

PEP ...

CXC is moving to digitally delivered exams. Eventually students won't need to travel to exam centres to do exams. You can stay home with your smart device and do your exam.

We can have a master teacher teach the whole country and help to address the "teach shortage". Campion College is leading the way in flipping the classroom.

Addressing Patrick Hylton "I expect NCB to match some of the things I'm doing in the education system"

These strategies form the [foundation] that will help the ministry to address the needs to meet the needs of the labour market.

Not just catch up but leapfrog!

We were leaders in the 1960s and we have the opportunity to leapfrog ahead of Estonia. We don't want to just be like Estonia, we ant to be better. Most education systems are based on models implemented over a century ago. Digital transformation is causing… new jobs requiring skillsets.

You will soon benefit from the new Jamaican creative minds.

Sign up for my upcoming Plone 5 Book & Video tutorials

plone 5 for newbies book and videos