Since you don’t want to give up on writing your memoir because it brings up troubling thoughts and painful feelings (unless, of course, it does prove too difficult and dangerous an enterprise), you will need to think through what you mean to do when those troubling thoughts and painful feelings appear. You might want to read in the trauma literature, the post-traumatic disorder literature, or the addiction recovery literature, as those are places where a lot of thought has gone into how to heal the past and how to deal with the past in the present.
If you intend to write a memoir, your journals are a gold mine and a treasure trove.
But they also come with shadows, demons and difficulties. Your hope may be that you will simply sit down with your journals, read them, and mine them easily and efficiently. But it is rather more likely that there will be more to the process than that and that certain inevitable practical and psychological challenges will arise, stalling you and even pressuring you to put your memoir aside. Get ready for those challenges! If you intend to make your voice heard, you must meet those challenges. Your journals are a wonderful raw resource; then the real work begins.