00;00;10;07 - 00;00;43;25
Jasmine
Hey there and welcome to this episode of Psych Attack! I'm Doctor Jasmine B MacDonald's. Lately, I've been thinking a lot about a specific area of research and. Hit a point where I really needed some back up. I've been thinking a lot about the organized, disorganized typology of serial sexual offenders or serial killers. If you've watched things like Criminal Minds, if you've seen the Hannibal Lecter films or you've watched, the Netflix series, Mindhunter, then you probably already know what we're talking about.

00;00;43;28 - 00;00;58;27
Jasmine
So today I've brought along a colleague of mine, a biostatistician, and all round excellent human, doctor Monsurul Hoq, to hack on the statistics of this. Work with me. So, Mons, thank you for joining me today.

00;00;58;29 - 00;01;22;19
Monsurul Hoq
Thank you, Jasmini, for this opportunity to, discuss about a stat. I mean, stat is, one of the things that, this is, is a fascinating is something that gives me insight in a lot of different topics and, and, being a biostatistician and, it started my career in studying education, then move to, public health and sociology.

00;01;22;19 - 00;01;47;00
Monsurul Hoq
Now, talking about you know, this topic that, you know, the ACLU learned and offenders. Yes, that's probably an uniqueness of it. But a study, it allows you to, work in different topics, and analyze the data and see the world, from a statistical point of view. So, that's probably what I enjoy. And that's why I become in bus station.

00;01;47;08 - 00;02;21;04
Monsurul Hoq
I'm originally from Bangladesh. And, studied my, on US masters there. Then, move, to Africa, work there as a bus decision, then move to Australia in academia for a bit. And my PhD. Yeah. So I guess, both from learning experience, traveling, working in NGO and on NGO academia. Yes. I've probably not, pure biostatistician anymore.

00;02;21;06 - 00;02;35;14
Monsurul Hoq
And by a biostatistician, I probably, as it is, my lived experience, experience, through different topics. And I love to see, that, proper biostatistics is applied in decision making.

00;02;35;16 - 00;03;07;04
Jasmine
Yeah. I'm really keen to hear what you think about this paper on that point. Before we get there, though, I wonder, well, firstly, as someone trained in psychology and social work who has done through that process some statistics and has always used it as a tool in, in specific topics that I'm interested in, I actually hadn't sat back and thought about the beauty of that, being a statistician or a biostatistician and and that being the lens to then have access to all these different topics.

00;03;07;04 - 00;03;11;28
Jasmine
I, I was getting goosebumps. When you describe that. That's pretty powerful. Very cool.

00;03;12;01 - 00;03;41;23
Monsurul Hoq
It is. It is. Yeah. I mean, as a biostatistician, or a statistician in general, like, you get to work with, lots of exciting researcher people from different, phase experts. And one of the, fascinating things is, you get to see the results before the war, and I have actually seen it. So, you know what the ins and outs of the data and and also this exploring the unknown all the time.

00;03;41;23 - 00;04;14;01
Monsurul Hoq
Like, I think that's probably one of the thing that as a citizen, I enjoy because, throughout my career, I've worked with lots of research. And then we come with some exciting question, or hypotheses based on their working experience. And, either we will collect data or we'll have secondary data and, and they will help me understand what is it that the one to find out or what is this question that they have, and using the data, you know, through the process that I have also learned that okay.

00;04;14;01 - 00;04;44;05
Monsurul Hoq
So sometimes I feel like, I mean, public health expert who knows about disease prevalence and incidence and what, what kind of activities might work. I'm not saying I'm an expert on those, but but, like, you know, you are having those insights, you are working with, those, experts on those fields. Yes. And I also like, during Covid, I was working on the vaccine acceptance, vaccine hesitancy and all this thing.

00;04;44;05 - 00;05;15;07
Monsurul Hoq
And, yeah, I was, lucky to have all this insight. And and I worked in a study which, worked on, with the Australian, people who will accept the vaccine even before vaccine was invented. It was back in 2020, in April, and 86% of Australians said yes. And ultimately that's the that's the figure. So, you know, that's kind of like the of being a sometime you were also thinking about causality something you were prediction.

00;05;15;09 - 00;05;34;23
Monsurul Hoq
Yeah. So it is exciting. Yes. But you are also not the face value. So this is something that we have to accept that as an a statistician you're probably not the face of that researcher. You're probably the probably the in engine, probably somewhere one of the, one of the main equipment but not the face.

00;05;34;25 - 00;06;04;03
Jasmine
Yeah. Yeah. That that scoop of knowing the results before anybody else does, even if you're if you're not the face of the project, you don't necessarily leading it. You still you know first. Yeah. So so instrumental is it. I just want to unpack very quickly. The difference between a statistician and a biostatistician is biostatistician. Is this really what you're saying when you say you have, knowledge and experience around public health?

00;06;04;03 - 00;06;06;10
Jasmine
Is that what we're talking about with the difference there?

00;06;06;12 - 00;06;36;10
Monsurul Hoq
Yes, yes. So, statistics is the core. And, pretty much you all learn the same, but then once you go into an application, obviously there's Business Step who deals with certain assumptions. The statistical test, when you go into public health or clinical epidemiology, you are more dealing with risk and then incidence prevalence and and again different types of statistical method that you were applying.

00;06;36;11 - 00;07;05;16
Monsurul Hoq
Then also there is actuarial demography. So they are all different you know subdivision of statistics to statistics. And then obviously you go into those discipline, you learn the terminology, you learn different mechanisms. So yeah yeah biostatistics is one of the subgroup is the main step. Yeah. But fundamental things are still same. We still talk about central limit Turing theory assumption regression model.

00;07;05;25 - 00;07;10;18
Monsurul Hoq
So the fundamentals are still the same. It's the tools that different.

00;07;10;20 - 00;07;12;25
Jasmine
It's just a subsection of the club.

00;07;12;28 - 00;07;13;14
Monsurul Hoq
Yeah. Yeah.

00;07;13;15 - 00;07;38;28
Jasmine
Yes. That's well, today I, I really wanted to pique your brain about one of the first papers that was published by this team that, you know, as I mentioned in, in the intro of the episode, the idea of organized, disorganized typology of serial killers is relatively well known. It kind of has had this Hollywood effect where it's popped up in all these film films and TV shows.

00;07;38;28 - 00;08;02;19
Jasmine
And I think partly that's because the FBI agents and the academics who are involved in the original work write a whole bunch of books and they've, you know, consulted on different TV shows and things. But I want to go back to the original work that all this is based on and, and kind of get a sense from you of what, what things, interesting or well done about it, whether there's any concerns from you.

00;08;02;19 - 00;08;26;16
Jasmine
And I guess I want to call out upfront, there was some concerns for me when I was reading it. I've always been really interested in this typology, but I'd never kind of going back to the original works. And the work was originally published for the Department of Justice in the US. But then the authors published a handful of peer reviewed journal articles, but only one that kind of talks, you know, kind of includes statistics.

00;08;26;19 - 00;08;50;22
Jasmine
And it's called sexual Killers and Victims Identifying Patterns Through Crime Scene Analysis. And this is a 1986 publication. I'm going to give a little bit of a background, and then I'm basically going to pass over the months and, and also lots of questions. And the listeners and I will we'll learn in the process. But I guess for folks who aren't across this yet, really, this is, one of the major turning points in criminal profiling.

00;08;51;04 - 00;09;22;22
Jasmine
And the idea, I guess, at the time, as described by Ressler and colleagues, is that prior to their work, psychological profiling really focused on the personalities of offenders. And I guess, one of the big influences originally was like psychodynamic theory, very Freudian. And they these FBI agents kind of took exception to that idea of trying to understand what's in the minds of people and work from trying to understand the psychology, to then apply that to a crime scene.

00;09;23;04 - 00;10;01;13
Jasmine
And also really prominent in criminology at the time was looking at the role of the victim and basically what it was about them that precipitated the events, or the, the crime, which is pretty problematic. I think that there's a big shift now away from that kind of victim focus of victim blaming. So what they did was trying to look specifically at crime scenes and, do analysis to see if there was a, if there really were differences between what they had developed through years of working on serial, offenses that hadn't been solved yet.

00;10;01;13 - 00;10;28;29
Jasmine
And they kind of realized through experience that there was this, this disorganized group, and there was this organized group of serial offenders. So your organized group tend to plan. They tend to be more methodical. Crime scenes are more controlled and less evidence is left behind. And then we have disorganized offenders who, kind of less planning is involved, more spontaneous, crime scenes tend to include evidence.

00;10;29;10 - 00;10;58;02
Jasmine
There's a whole bunch of characteristics this paper talks about them of of, what is different between the two, in this typology. But I guess I want to hand over to you, I'm going to reflect a first broad point and just see where you take it. Simmons. And the first is this idea that while it's a very data driven method of looking at a crime scene, they are first coming to it with their own kind of preconceived categories or IDs and that was something I found kind of interesting.

00;10;58;05 - 00;11;02;19
Jasmine
So I'm going to stop talking and post you see what you think.

00;11;02;22 - 00;11;31;24
Monsurul Hoq
I mean, I read this report with interest, and of course, the first issue is that not and I'm not an expert in this area. Then this. So that was, one of the interest is to really understand, the standard was used in, in criminal psychology. I think the first point that, as you mentioned, really struck out for me is the positive is that they wanted to describe these things from the offender perspective rather than victim blaming.

00;11;31;29 - 00;12;02;19
Monsurul Hoq
I think the paper sets up in the first paragraph saying that the victim are the cause. So that's kind of the narrative they wanted to change. So I did have to call out for that. Like this is really, really good initiative in 1986 that you're talking about. So in the current now in 2025, I guess many of the research that we are doing where we really wanted to talk about the research driven from a perpetrator perspective rather than victim blaming, I'd also like to change those narrative, like even if we don't have access to the data from the perpetrator, we do.

00;12;02;22 - 00;12;25;01
Monsurul Hoq
If if we have analyzed the data for for the victim, we do not say this is the this is the course victims of the call. So I guess yeah, that is part of the strength of the paper. When I read the paper, I think it didn't really actually struck out that the they did the categorization beforehand. Actually, this is what I would suggest that you do, because these are the expert people who actually work on that field.

00;12;25;03 - 00;12;47;20
Monsurul Hoq
They have experience working on this area. So they definitely have hypothesis. And based on that, they have decided that this is organized. And this is do recognize what would be interesting to actually see after the data analysis. Do you still believe that their hypothesis was correct or not? So that's probably where I was like a little bit disappointed.

00;12;47;20 - 00;13;04;24
Monsurul Hoq
I really wanted to see what was the justification that they put forward, the justification that this is the why we have categorized them into group. We are going to look into all these data and then come back and see whether our hypothesis was correct. And this is as I said, like, you know, working in public health, working with the clinician.

00;13;04;24 - 00;13;26;00
Monsurul Hoq
Many of the cases this is this was the case, like people actually worked in this field. They come up with that. I think this is what it is, but I don't have enough data to prove it or show evidence. And then then that's where we designed this study. So, when I was reading, one of the review, I thought, oh, that whole concept of categorization was criticized, which is really interesting.

00;13;26;00 - 00;13;53;27
Monsurul Hoq
And I found was not surprising. So I think this is where, subjects of the discipline, the discipline, the things that you apply varies. And not everything is, black and white. Right. So obviously that would be a gray area. And categorization is is one thing, you know, where do you put the cutoff even in, in, in, in blood test.

00;13;53;27 - 00;14;03;01
Monsurul Hoq
Right. When you do a blood test, you go to a GP and then say, you know, things are normal or abnormal. But if you really look into those data, these are also based on cutoffs.

00;14;03;03 - 00;14;04;02
Jasmine

00;14;04;04 - 00;14;28;06
Monsurul Hoq
So something can be individual as well. So I'm just saying like you know this whole talking about this categorization. So, so for me that didn't really surprise me. I was really like okay fine, we have done it based on their experience. Then they have selected 36 sample obviously as a not enough sample. But also we are talking about, you know, so I don't know how many events they actually had.

00;14;28;08 - 00;14;49;00
Monsurul Hoq
I probably say that I was looking for a little bit more information that they had, let's say, you know, these many cases of them, they selected 36 or they had 36 cases only. That's what they have analyzed. So that is one area. I mean, like if you're analyzing administrative data and you only have access to statistics, then this is what you have.

00;14;49;00 - 00;15;02;06
Monsurul Hoq
This is what you analyze. But if you if you have let's if they're FBI, they might have access to 1000 dataset. And they have analyzed 30 things. So that two are completely two different scenario. And we don't know about that.

00;15;02;09 - 00;15;06;01
Jasmine
Yes. So they the total samples 36.

00;15;06;01 - 00;15;06;23
Monsurul Hoq
Yes.

00;15;06;26 - 00;15;35;25
Jasmine
So they were touring different parts of the US as federal agents, visiting local stations to help with cases, but also to do training and different things. And along the way, where possible, they would visit local prisons and any serial sexual offender that they could or serial killer that they could get access to. They interviewed so it's one of the kind of main criticisms that comes out of this work is it's not randomized.

00;15;35;25 - 00;15;58;01
Jasmine
The sample is very small. And the way they interviewed wasn't standardized. Now we we don't know what the population is like. There's no way to know what the population of serial sexual offenders is. They're all male because there were because there are all male prisons. But also at the time, it wasn't really accepted that women would be serial offenders.

00;15;58;04 - 00;16;22;04
Jasmine
So I don't know that that bothers me too. And I was really kind of curious to see what you think about that, that random, all these things that are important statistics, but also the practical aspects like if this is who they have access to and we don't know what the whole population is to sample from, I feel like that they've kind of gotten a bad rap for doing something that's quite obviously these limitations, but is quite innovative at the time.

00;16;22;06 - 00;16;41;25
Monsurul Hoq
Let's say when you're saying this is a people that they have access to, and I'm just getting this as an administrative data, something like this is what we have, right? So in that case is completely fine. That's the 36 they had access to. Now the question is then the generalizability of those. You cannot question that a license because that's fine.

00;16;41;25 - 00;16;55;28
Monsurul Hoq
Yes analysis can be done. But how much can you actually interpret the findings to generalize all the seriously are all kind of have, you know, validate the idea that is this categorization correct or not?

00;16;56;00 - 00;16;56;22
Jasmine

00;16;56;25 - 00;17;21;00
Monsurul Hoq
So that that that's probably is a question. But I guess when when they were explaining the study, they said it's a descriptive study, was just explaining what we have found based on the 36. I guess that's probably the limitation. They were really open about. But how people will read and interpret is not in their head. But I guess that's probably something that we all have to, you know, deal with any record.

00;17;21;00 - 00;17;43;07
Monsurul Hoq
Amia like, you know, you do a very good study. You might not know that the result of the way you actually wanted to interpret will be interpreted. I mean, in the in the current era, we have knowledge translation. We have we have all the other things that we have now, like where there are a lot of other other disciplines, help us to really communicate the matters from the research.

00;17;43;09 - 00;18;10;20
Monsurul Hoq
But I'm not sure that that was the case in this case research. I mean, maybe it was just a thought provoking research, like there's supposed to be only for giving the idea. And and based on the of the review that I have read, maybe it was actually served its purpose that it actually came up with an idea. People have counter that idea, but it actually led into something constructive.

00;18;10;22 - 00;18;13;29
Jasmine
And that's science, right? We have to stop with.

00;18;14;01 - 00;18;35;20
Monsurul Hoq
Yes. Yeah, yeah, yeah. You cannot always do a gold standard research when you cannot do a randomized controlled trial for everything. Sometimes you do have to deal with the data that you have. So in that case, yes, there are some question, but I'm not saying that they could have done a random sample because they might not have access to all this data.

00;18;35;22 - 00;18;37;05
Jasmine
Yes.

00;18;37;07 - 00;18;59;27
Monsurul Hoq
And also they they actually interviewed those offended them. The people might I mean, some of them might be randomly selected and they would choose not to interview. So in that case, let's say you actually end up with the same 36 after doing all the randomization. So still have the same output. Yeah. But I guess yeah it's the approach that you take.

00;18;59;27 - 00;19;21;20
Monsurul Hoq
So yes I see you're absolutely right. Like it's a practical level. Like you know maybe it wasn't possible for them to do a randomized sample. Or maybe they had a idea and they thought, let's look at this with the 36 and pilot it, test our assumption whether it is correct or not. So that's probably one. Yeah.

00;19;21;20 - 00;19;58;14
Monsurul Hoq
But thanks for clarifying that. What was done in 1986 reporting P-value. Only naught providing summary statistics for all the comparison only for those significant one was acceptable in 96 is still acceptable to some journals nowadays. But majority of the, academics, who work in the statistics probably don't like to base all the inference based on p value alone, and also p value less than 0.05.

00;19;59;08 - 00;20;21;22
Monsurul Hoq
I think few years back there was a, paper in nature where I think more 500 a statistician sign that let's not make a decision based on p value less than 0.05. Yeah, that is also an categorization, right? We are making a big decision that this is significant. And this is not significant based on a random cutoff.

00;20;21;24 - 00;20;22;22
Jasmine

00;20;22;25 - 00;20;46;29
Monsurul Hoq
But I would say that in the table one, they did put the p value of those which are very close to 0.05. The Did report, those who are .109. So they even though in the method they say all those which is less than 0.05 is significant. The difference in those which are close to 0.005. So that's probably good.

00;20;46;29 - 00;21;13;29
Monsurul Hoq
Like they did not completely reject those which are .0607 because they were actually not less, but it's still there. So that's probably the p value in terms of the actual analysis. It mentioned that they did t test. The paper mentioned that t test. Most of the variables are categorical. There were some which were continuous and they reported mean for categorical variable.

00;21;13;29 - 00;21;38;19
Monsurul Hoq
Obviously you're calculating proportion. So they did report the percentage. And then we did parenthesis the number. So in that case I guess the best bet is a kind rise test a proportion which actually looks into a z distribution not t distribution. So that's probably something that I was not sure. We will probably go with a more than practice.

00;21;38;21 - 00;22;02;09
Jasmine
That's a really interesting point that I want to understand better, because the analysis is we've got 36. This is just to bring folks up to speed who don't know the paper, the 36 offenders. But then to be a serial offender, you need three or more crimes. So what they're comparing is the crime scenes, the individual cases across those 36 offenders.

00;22;02;12 - 00;22;13;18
Jasmine
So that's a little bring folks up to speed. But then to back to what you're saying, talk to me about talk to me about that difference that you, you mentioned there. I don't quite get it. And I want to.

00;22;13;21 - 00;22;42;16
Monsurul Hoq
So I and they have a in the metal. They talked about all those variables like more than 300 variables from four different data set. And in the table they have provided those variable names and in, in some context variable labels. What I understand majority of them are categorical variable. Yes. Where you can see whether the someone probably within a certain age group.

00;22;42;18 - 00;22;45;00
Jasmine
Like left footprints at a crime scene.

00;22;45;02 - 00;23;23;21
Monsurul Hoq
Exactly. Or, or use a vehicle and not used a vehicle, something like those. Also, they had some scale which is continuous, like between 1 to 10 or more like a ordinal scale. But let's say for the example for this, this kind of bunch of is type like the number in my practice, if I'm analyzing categorical data, and especially in this case, comparing the proportion between two variables, I will do a test a proportion to see whether the difference in proportion, what is the difference in proportion and the confidence interval, and see the variability.

00;23;23;21 - 00;23;47;09
Monsurul Hoq
And then obviously the p value. But this paper talks about details. So details is a distribution obviously. And you have to have an independent normally distributed. I don't think you can easily check those based on a categorical variable. But for large sample that's all normally distributed. So let's not let's, let's assume they are for large sample.

00;23;47;09 - 00;24;10;28
Monsurul Hoq
But this is not large. And then the t-test is usually used for comparing means. But you know you're doing proportion. So what we usually use test of t proportion a test of two proportions which usually follows a check distribution. Before coming to this talk I did actually look for what your distribution first invented. It's like 1912 or something.

00;24;10;28 - 00;24;16;12
Monsurul Hoq
So obviously z distribution was the year in 1986.

00;24;16;15 - 00;24;39;02
Jasmine
And this is a thing I've been thinking about too, because, the stats in this, by De Gastineau, who was a biostatistician with PhD from Harvard. And so I'm thinking like, this is a person is very knowledgeable and experienced. Were these other methods or whether or not the methods had been invented, were they in practice at the time?

00;24;39;02 - 00;24;43;12
Jasmine
You know, this is always tricky when we're going back 40 something years.

00;24;43;14 - 00;25;10;06
Monsurul Hoq
Yes. And the other factor is also, you know, there's there's not enough sample. So you could have done parametric non-parametric as well. So you could have done chi square test when you're doing the, proper test of proportion or you could have done a lot of other things like different approaches. So yeah, that's probably another thing that I was like, why is why is D not other things?

00;25;10;08 - 00;25;38;11
Monsurul Hoq
Maybe there is just variation, but probably with my practice I'm not aware of, but maybe there is an alternative, justification for, those who work in this field or those who know more about the teachers. Probably possible. And get this details to some extent. I'm not going to say they're similar, but, you know, the underlying options are there, there.

00;25;39;09 - 00;25;45;13
Monsurul Hoq
So I'm not justifying what they have done, but but I'm saying there could be a justification that I'm not aware of.

00;25;45;15 - 00;25;58;08
Jasmine
And that full report of analysis is actually not probably there is some library in the US that I'll eventually access that report from, but it's not. In the few months I've been looking, I haven't just found it online.

00;25;58;10 - 00;26;27;03
Monsurul Hoq
And that was actually one of the thing, because they did say that they published the whole justification of that analysis is published. And and I wanted to have a look at that one, but there's no such thing. And that's where like, you know, in the current era era, we talk about open science, like where you can actually see all these things nowadays, even there's, request for data to be available so that other people can analyze those things, which is not there back then.

00;26;27;03 - 00;26;51;04
Monsurul Hoq
Right. You're just only reporting p value and saying this is significant and that's it. Then they have provided a summary statistics of those which are significant. But no not those are those. So as a reader you are relying on them a lot. I mean of course that's that's what academia is at the moment. Like you know, we of course rely on the researcher.

00;26;51;05 - 00;27;23;13
Monsurul Hoq
But at the moment there's the peer review, there is all these open access. Yeah. There's all these, criteria that you have to follow before you publish a paper, which actually gives you confidence in those kind of analyzes, whereas in this case, probably not. One other aspect is from the 36 offenders. They have, collected data from 100 plus, victim.

00;27;23;16 - 00;27;57;23
Monsurul Hoq
Now that means you have longer dependent data. Basically for some individuals, you have multiple victim. But when the victim cases were analyzed, they were analyzed as an independent characteristics. So I'll start in. So serial killer might have a preference for those that were victim for that particular serial killer. So I think that another approach like, you know, in analysis got clustering basically.

00;27;57;28 - 00;28;33;15
Monsurul Hoq
So because that's whoever the victim of that is constitute a cluster. So that another aspect was not considered in analysis. I would have done that to understand because the motive may have been different by serial killer. And obviously we were also not provided with any numbers. Is that between the organized and disorganized number of victim, whether it is different if, if someone is disorganized, serial killer.

00;28;33;17 - 00;28;53;24
Monsurul Hoq
I mean, I'm not an expert in this area. What I was just conceptualizing is that if somebody is disorganized and, and as the this paper is saying that somebody acted because of that location, then how come that person is a serial killer? Because if you are there, there should be some motive, right? That linking those incidents.

00;28;53;26 - 00;28;54;14
Jasmine

00;28;54;17 - 00;28;58;18
Monsurul Hoq
So that is not talked about in this paper at all.

00;28;58;21 - 00;29;12;28
Jasmine
No. And some of the behaviors that they talk about or compare with these t tests, the frequencies can be as low as four. So it can be just as you say, be the one person and no representative of one group compared to the other.

00;29;13;01 - 00;29;44;18
Monsurul Hoq
Exactly. Yes. So that that is one of the thing that, I was not sure sure about when we looked at that one. I think in general, the presentation obviously there is that. And roughly across the four, data sets, I think 139, 100, it is 1157. So there's roughly 350 variables that they have analyzed. And, 50% of them were significant.

00;29;44;18 - 00;30;08;23
Monsurul Hoq
So you got about 20% of significance. So it's pretty much that's where I guess nowadays you're talking about like a p hacking. So you're doing all these analyzes without justification of, you know, of course, in 1986, the computer wasn't that fast at the moment. You know, computer nowadays is very fast. You can pretty much do a lot of analyzes.

00;30;08;26 - 00;30;44;04
Monsurul Hoq
And that's where, we are continuously suggesting people or recommending people that, you know, please have your analysis plan first, really conceptualize it first and then going to analyze it. Don't look at your data. Don't peek in the analysis first. So yeah, out of one out of 250 variables if the are significant is not surprising. And I was looking at one of the variable which was they call it, you know, sexual concern.

00;30;44;07 - 00;31;06;08
Monsurul Hoq
So out of the five variable, only one is significant. And that's what they reported. But that is part of a construct. Right. So there is five variable. One of them are significant. And they those are like out of total number out of 17 in 1 group and eight in another group. And then out of the 17 you have 12%.

00;31;06;08 - 00;31;33;19
Monsurul Hoq
Out of the eight you have 62%. So that's another area of concern. As an a statistician, when you just looking at that one like you have tested five variable, one is significant and that's what you reported nothing about the other four. Maybe you would have seen the same pattern, but just because of there is not enough number, you're not seeing a significant result.

00;31;33;21 - 00;32;03;26
Jasmine
There are a bunch of like like subscales like that in like in one of the tables there's torture one, four, five and nine. So presumably somewhere there's two and three and six, seven and eight is really interesting because most of what's reported is significant. As you say, there's some that aren't. But then most of the ones that aren't there approaching a .05, is it fair to say what we're talking about here is cherry picking the things that tell a story for publication.

00;32;03;29 - 00;32;21;11
Jasmine
And arguably on the one end, it's on the most honest utility based, and it's trying to present the things that could be used for a profile. But it's also problematic because of the number of analyzes we do to get to these points. Like so this we're talking about the idea of out for inflation.

00;32;21;14 - 00;32;49;20
Monsurul Hoq
Yes and no. Yes, they have done a lot of analysis. And obviously that's where, as I say, like an hour or 250 variables, 50 of them are significant. Yes. So there is also inflation. But also on the other hand, I would have been concerned if only 5% of the analyzes were significant and then only describing those so that have 20% of the significant variable.

00;32;49;20 - 00;33;11;14
Monsurul Hoq
So that is something they are not only presenting or making a story out of nowhere. So there is some thing there's some findings. So I think the cherry picking or the packing, the concept that came is that when you actually don't have anything of yours, slicing and dicing your data in different ways, to really find something.

00;33;11;16 - 00;33;20;26
Jasmine
They say. So this really is a it's nuanced. It's a blurry area in an innovative and high stakes public health issue.

00;33;20;29 - 00;33;46;24
Monsurul Hoq
Okay. Yes, yes, yes. So I'm not going to penalize them for that. I mean, because they have reported the actual p values, they are not, so they have done that. That is more than enough significant findings. If there was only ten out of the 250 analysis. Yes, I probably said they are looking at p hacking or cherry picking or other inflation.

00;33;46;24 - 00;34;09;11
Monsurul Hoq
They're not doing that. They, they reported more than enough. Yes. Where I'm calling out, that means that they should have presented to all 350 analyzes. And if we do that, in that case, you probably have seen that. And the actual concern, those which are missing out are also trending in the same direction.

00;34;09;14 - 00;34;10;07
Jasmine

00;34;10;10 - 00;34;32;24
Monsurul Hoq
So even though they might not be significant, but the actual effect size, the difference is between organized and these organized were probably trending in the same direction like those which are significant. So in that case what would have happened is that even though they are not significant, as in reader, you are making more conscious decision. You're saying, okay, no, I can see that this is purely numbers.

00;34;32;24 - 00;34;41;03
Monsurul Hoq
They don't have enough numbers. That's why those two are not significant. But I can see the behaviors all tending towards the same direction.

00;34;41;06 - 00;34;42;18
Jasmine
I see, I see.

00;34;42;21 - 00;34;48;17
Monsurul Hoq
I think that those would actually help in favor of their argument.

00;34;48;29 - 00;35;08;15
Jasmine
Yeah. I'm seeing that now through your description. If they had a sample of a thousand offenders and then all of the crimes for those people, then I guess that's more what we'd be thinking of for inflation is the more tests and you've got a really big sample, you're gonna find something so actually works in the other way than I thought.

00;35;08;18 - 00;35;35;10
Jasmine
Yeah, this is why we talked to people who have specialist knowledge. Because I was looking at when I was thinking, like you, I wouldn't I don't know how it would work, but I imagine you could do another type of analysis like, I'm going to say some stuff and you're going to roll your eyes, but like, you know, I'm thinking of things like a factor analysis or structural equation modeling or something where we can do a bunch of analysis at once that can reduce the number of tests.

00;35;36;00 - 00;35;39;07
Jasmine
But I hear what you're saying. It's really interesting.

00;35;39;09 - 00;36;15;15
Monsurul Hoq
There's two point that you made. And, you know, as an analyst, we analyze the data. The findings are the findings, and the interpretation is always subject, and it depends on the knowledge of that topic, how expert you take, you know. So the same findings might for a statistician might look okay. It's not important for me. But then when the expert looks at it then I'm like okay no this is really important findings and and and I'll give you a two example from my own experience.

00;36;15;15 - 00;36;38;16
Monsurul Hoq
And so, when I was working in, in, clinical research, there was, and I said a single significant finding. I'm not going to more detail into about that, this research, but, it is about a procedure. And then after the procedure, the, the clinician hypothesized that they have similar to this study, they have done two different types of procedure.

00;36;38;16 - 00;37;12;13
Monsurul Hoq
And they said, well, let's hypothesize that if the outcome of the two procedure varies by this much, it would be of interest. There was enough sample and the findings were statistically significant. And I was excited. Whoa. There's a statistically significant finding right. But the thing was the confidence interval this week. Calculate where all ten. And it was, a very narrow because you have enough normal sample size.

00;37;12;16 - 00;37;23;26
Monsurul Hoq
And it's really consistent, which is all good. Good thing about a single license. Then we presented to the clinician and they said these findings are not particularly meaningful.

00;37;24;14 - 00;37;27;24
Jasmine
What is the practical significance exactly.

00;37;27;24 - 00;37;55;27
Monsurul Hoq
Yes. So they were actually happy. And both of I mean, it was really, a learning for me. They were happy because what it means that irrespective of whichever procedure you apply, the clinical outcome doesn't vary that much. Yes, there is slight differences. It is statistically different, but it shouldn't decide which procedure you apply.

00;37;56;00 - 00;37;56;21
Jasmine

00;37;56;24 - 00;38;28;06
Monsurul Hoq
And then it really went into a practical because you have to have those equipment. You have to have all those scenarios to really do those two procedures. So then the doctors were really relieved like okay fine it doesn't really fit. In another scenario this is during Covid. I was analyzing data for some hospital cases and, and the I just did the descriptive study and the clinician was really like, what is the p value in this paper?

00;38;28;06 - 00;38;52;03
Monsurul Hoq
What is the p value in this paper. Like you haven't done any p value. And I said to to the clinician and to this, this part that we do p value and all this statistical analysis for sample characteristics. So you have a population. You take a sample random sample and you try to make an inference that this is my findings that is generalizable to the overall population.

00;38;52;06 - 00;38;58;10
Monsurul Hoq
But you actually have a population because during the early days of Covid there are not many Covid cases. Right?

00;38;58;13 - 00;39;02;02
Jasmine
Yeah. You're and it's a it's a capital N.

00;39;02;04 - 00;39;11;03
Monsurul Hoq
Yes. Your ad is actually capital N so you don't need any p value. Whatever you are finding easier finding you don't need to justify.

00;39;11;06 - 00;39;20;08
Jasmine
We don't know the population size. Like I like incarcerated serial killers in that region. This could be a capital N yes.

00;39;20;10 - 00;39;46;17
Monsurul Hoq
We don't. Yes this could be that thing. Then also, that I was talking about the focusing on statistical significance alone and not reporting those other insignificant findings, the effect size the differences would have if I was an expert in, in, in criminal psychology, I would have known that, oh, is this difference large enough or is this difference consistent across multiple outcome?

00;39;46;19 - 00;40;05;05
Jasmine
I say so there is professional an expertise judgment that's being made here. And undoubtably this group of experts in this area. And presumably also there's limitations that the journal have put on them of what they can report and include as well.

00;40;05;08 - 00;40;06;02
Monsurul Hoq
Yes.

00;40;06;05 - 00;40;13;03
Jasmine
Interesting ones. And also, I want to pause a beat. Yes. That's that. Please keep going.

00;40;13;05 - 00;40;46;15
Monsurul Hoq
And it also like you are actually like in those days, those, supplementary data wasn't available. Like you, nowadays you can publish thousands of data as a supplementary to the main channel. So this was not possible at the time. So I guess probably that's where, those author couldn't provide all those analysis. But what I'm saying is that these are the things that you have to consider when you are in, in the, in, in current era, like publishing this paper as it is probably will not be acceptable anymore.

00;40;46;18 - 00;41;08;22
Monsurul Hoq
You probably have time to publish all those things. You should describe what does the difference between organized and disorganized means? What sort of difference you are looking for? Is it a difference of 20% of certain characteristics that it will say, okay, this difference is large enough for us to say that there is difference or is it 5%?

00;41;08;22 - 00;41;09;11
Monsurul Hoq
Yeah.

00;41;09;14 - 00;41;48;06
Jasmine
Because it context wise, at the time of this, when the work was being done through the early 80s, and then this is published in 1986. At that time, there was a real peak in serial offenses in the US. So do you think, you know, it's hot? It's all hypothetical. But I do question whether this could be published now if if we had no knowledge in this area, even though we've got these different, requirements of open access data or whatever else, if it was such a public issue that policymakers and the public was so concerned about, we knew nothing about it.

00;41;48;08 - 00;41;51;20
Jasmine
Do you think actually this level of analysis could get published?

00;41;52;06 - 00;42;12;21
Monsurul Hoq
Yes. In that context, if we don't know anything about it, if this is something and yes, as I said at the very beginning, that this is probably a thought provoking paper. That's probably what the authors thought. Yes. We are not 100% perfect. Yeah, but this will start a tsunami. And that's what happened.

00;42;12;23 - 00;42;35;21
Jasmine
And it did. Yeah. And which is a big part of why I wanted to speak to you, because this isn't your area and it did start a tsunami. People are still really passionately arguing for or against this model. So to have someone come in and just look from a statistical methods and statistical issues approach has been really nice.

00;42;35;24 - 00;42;36;27
Jasmine

00;42;36;29 - 00;42;57;16
Monsurul Hoq
And there's another point that you mentioned is that, you know, I would come in to tell you a lot of fancy analysis technique, right? As a statistician, I mean, biases. So best bastardization. I think this all the time that we do and something like this. And then when you go to the review and something like that, and someone else will say, oh, let's do it in a different way.

00;42;57;18 - 00;43;25;08
Monsurul Hoq
Yes, you could do that. You can always analyze more fancier way, but you have to ask, what is your objective? What is the which is this question that you are trying to answer? How do you time? I mean, if I factor if it was yes then yeah, let's let's do factor analysis. But I think the author here was really interested to do two objective.

00;43;25;08 - 00;43;51;08
Monsurul Hoq
First one was profiling. So basically just describing all the profiles. Second one, the talk about predicting, which I do not think they have done in this, because then you really need to go into prediction modeling. Okay. Let's if we do input the victims, this is these characteristics that will tell us the likelihood of somebody experiencing that, that, thing.

00;43;51;08 - 00;44;16;28
Monsurul Hoq
But I guess that's probably that they didn't talk about they didn't talk about whether, if you put this, this, this offender characteristics that will tell you whether somebody is organized or disorganized. They didn't do the profiling. These are the characteristics, but they didn't really go into predicting. So that's probably one of the thing that but other than that yes I mean obviously you can do factor analysis.

00;44;16;28 - 00;44;41;24
Monsurul Hoq
You can do, principal component analysis. Those are things that that came into my mind as well. Like, you know, you can do sequence analysis to really understand how, certain profile is closely related. And then you can do a lot of fancy analysis, with current, you know, statistical packages in, in Stata. Yeah. They can do a lot of fancy things.

00;44;41;24 - 00;44;55;25
Monsurul Hoq
But is it your objective? That's probably where, you have to ask the question. I mean, and then sometimes we do get that, like, you know, you will, you will do an analysis and somebody else will say, oh, you could analyze this this way. Like, is this really my object?

00;44;55;27 - 00;45;10;12
Jasmine
Yes. And that comes back to what you said early on that they are the experts. And they had already identified these two groups were practically useful. It was coming out of the the cases they were working on. How would you have done the analysis.

00;45;10;14 - 00;45;14;03
Monsurul Hoq
Out of design a case control study?

00;45;14;06 - 00;45;15;28
Jasmine
Okay. Tell me about this.

00;45;16;01 - 00;45;56;23
Monsurul Hoq
What are the serial killers of the case? And I would define the control, which are not serial killer, but they are similar kind of case. Other offenders. And understand the differences of those two groups. And I think that their objective is really identifying the organized and disorganized within the serial killer. So having a control group. So basically they could be another offenders from the same jail.

00;45;56;25 - 00;46;02;08
Jasmine
But they might have only had one violent crime. So they're not serial offenders okay.

00;46;02;10 - 00;46;43;00
Monsurul Hoq
Yes yes yes. And then probably see a comparison how that works and how that is. So that's probably just putting a perspective. Because when you are seeing these analyzes your descriptive only describing it, you're not having a comparison group. So you have nothing that even if I know it's not an apple, an apple or comparison, but something even if you're comparing apple and orange, that's still like you are comparing, you are getting something in terms of the actual analysis, as I said, I've not provided a very basic descriptive table with all the details of every variable.

00;46;44;02 - 00;47;06;21
Monsurul Hoq
And provide how many were missing for each of the variables, something that they have mentioned in the paper but did not describe how many were missing. So actually put all those information out there, to test a proportion where it is categorical and then do, you know, mean test of equal in your mean. That's fine.

00;47;06;21 - 00;47;33;11
Monsurul Hoq
I would report the actual effect size confidence interval and obviously p value will be there, but I will not make my judgment based on the p value. I'll probably do something like a forest plot where, you categorize, okay, these are the common characteristics. Let's put them in one plot and see whether all the differences where the differences are, are they are actually clustering.

00;47;33;13 - 00;47;59;29
Monsurul Hoq
So that would be my first analysis. But as I said for the victim, all that I own have just for the clustering because that is there. And then I will look at some sequence analysis as well, just to understand, whether the motive has changed for serial killer from one offense to another.

00;48;00;04 - 00;48;04;21
Jasmine
So the sequence of the order that they've committed the crimes, if there's a chain.

00;48;04;23 - 00;48;30;02
Monsurul Hoq
Yes, yes. Yeah. But other than that, to address the objective, probably I would have done a predictive modeling as well because that's the second objective. But within the objective, that's. Yeah. It's not vastly different, but I think the presentation would be a lot different than what they have done. But but as I said at the beginning, I like that the this is the experience they have already described.

00;48;30;02 - 00;48;59;11
Monsurul Hoq
This is the organized and disorganized. So I'm probably going to provide more details about how they were selected, provide a little bit more justification on the inference, like not just the significant, honestly insignificant one as well, and describe them together like, yes, out of the five six concerns, three were significant, two were not insignificant. But as a construct, they're all trending towards something like that.

00;48;59;11 - 00;49;05;23
Monsurul Hoq
So it would provide more context to the, inference.

00;49;05;25 - 00;49;31;15
Jasmine
One of the things that I've noticed is when when I work with folks who are less experienced in the methods that I'm experienced in and they they're just fresh out of training, they are very black and white in how they apply things, and they like to follow rules. And so I think my reading of this, I'm not being an expert with statistics is there are certain rules I was looking at and thinking, oh, this is a problem.

00;49;31;15 - 00;49;53;29
Jasmine
But but actually a lot of what you've described has made me less concerned of this paper. But I just want to clarify a couple of specific points. So how concerned are we then. Oh, everything considered together the fact that these are experts who've done this work, that the size of the sample wasn't so large that we're really worried about the kind of analysis they've done.

00;49;54;02 - 00;49;57;13
Jasmine
How concerned are you that effect size isn't there?

00;49;57;16 - 00;50;24;06
Monsurul Hoq
I'm not that concerned because those which they have reported, the size is very big. So for a small sample size, that's probably given that you will only get significant where the things the differences have to be very big. Yes. Because you will not get a significant finding for a 36 sample. You probably not get a significant result for difference of 2% 3%.

00;50;24;06 - 00;50;54;07
Monsurul Hoq
They have to be something like 3,040%, which is what you have seen. But the real question is the generalizability of that difference, because you don't know how these 36 were selected. Yes, there is difference within these statistics. There is difference between organized and disorganized. That's fine. But can you take this finding and apply it to the all the serial killers?

00;50;54;10 - 00;51;00;14
Monsurul Hoq
That's probably where I probably say no, I'm not sure about that.

00;51;00;17 - 00;51;22;20
Jasmine
That's the key point here. Yes. Yeah. So what I did, because I'm a giant nerd and I have time on my hands while I'm focusing on this work, is I went through and put in a table each of the tests and grouped them by, you know, across the tables. They have t test not reported, so it's included, but they actually don't say what the t test was.

00;51;22;20 - 00;51;45;15
Jasmine
So we can't say whether it was significant. The ones that were non-significant then significant at 0.5. But because of all these things that I was concerned about, that actually, you're sharing your expertise. That maybe isn't a great issue. One of the things that I had had in my mind was maybe we needed a more stringent alpha value, like maybe it's .01 or 2.001.

00;51;45;15 - 00;51;59;13
Jasmine
And I kind of had blocked these to see and wanted to check with you. Would we necessarily be more confident as we get more stringent? But I think this conversation, what I'm learning is no like .05. We're pretty happy with that.

00;51;59;15 - 00;52;32;15
Monsurul Hoq
No, actually, I'm sorry that you got that. Actually, what I was saying is that we shouldn't be fixated on any cutoff for p values. We should report p values as it is. So in this paper they actually reported p values not only point five but also .09.06 that reported other p values as well. Even though they are significant and not, you shouldn't make your decision based on p value less than 0.05 or more than 0.05.

00;52;32;17 - 00;53;07;05
Monsurul Hoq
You should report p value as it is, and if it is 0.01 or like 0.0001 or something like that, what it means that you have strong evidence in support against your null hypothesis. Yes. But if you have 0.01, you still have evidence, but your evidence is weak. Yes, it's not like you have no evidence versus you have evidence because that's how we interpret significant and not significant.

00;53;07;05 - 00;53;24;23
Monsurul Hoq
I think the narrative is not like that. The narrative should be. I have a strong evidence, I have some evidence, have weak evidence. So you have to really interpret it more like a fluid like this. Exactly. The criticism we have this for this paper is that categorization.

00;53;24;26 - 00;53;26;02
Jasmine

00;53;26;04 - 00;53;59;16
Monsurul Hoq
In statistics we have the same concept. We are still we are debating that. Don't interpret your data based on p value, whether it is less than 0.05 or more than 0.05. You actually interpret your data based on your effect size, your confidence interval, looking at the context, you know, non-significant data, all those things together. You make an informed decision whether it is significant or not.

00;53;59;16 - 00;54;21;12
Monsurul Hoq
And also, one of the other thing that I mentioned is its clinical significance that that we use a lot in public health. I'm not sure that this is same thing exists in sociology or or psychology, but I think one of the things is the mean, is the difference practically meaningful? It may not be statistically significant, but the difference might still be practical, significant.

00;54;21;14 - 00;54;45;18
Monsurul Hoq
And that's where the case here, like, you know, in this paper, we haven't even recorded any non-significant difference. So you don't know whether maybe the differences were practically meaningful. But just because there's not enough number, it's not significant. So yeah, I mean, it is not cutoff. It is not p value is not binary. P value is a in a continuum.

00;54;45;18 - 00;54;48;22
Monsurul Hoq
So we actually have to analyze that in a continuum.

00;54;49;15 - 00;54;52;14
Jasmine
Thank you months. Really appreciate that explanation.

00;54;52;17 - 00;55;06;13
Monsurul Hoq
I will say it. You know one thing that the whole Alpha P hacking and foundation, all those things you actually started from, interpreting your data based on p value 0.05. I'm not.

00;55;06;16 - 00;55;07;10
Jasmine
Okay.

00;55;07;13 - 00;55;41;06
Monsurul Hoq
Because let's say you have analyze in this case 350 variables, right. And 20% of them are significant. In this paper. So 20% of your findings are significant. So it's kind of it's not by chance anymore. So that's 0.15 is really what we are considering is one out of 20. So it's really by chance if you're saying that okay I have done something and I found it one out of 20, you are really thinking, oh probably I got it out of chance.

00;55;41;09 - 00;56;14;16
Monsurul Hoq
Let's not really focus on it. So if you're getting it two out of 20, you're saying, no, maybe it's not even a chance anymore. Maybe there is some substance in it. Right. So that's where I guess the whole p value concept is. What we are saying is that if you're doing it 100 times and if you get four out of 100, you are saying, oh, is by chance, and if you're doing six out of 100, you're saying, oh, not there is actually something significant.

00;56;14;19 - 00;56;53;17
Monsurul Hoq
So difference is only two. So that's way like, you know, don't make a cutoff when this is in continuum. So yeah, I think that that if, if one message that I can take up from all this discussion probably is that is that, that don't rely on any statistical significance based on p value before the effect size. If you put the confidence interval and see whether that finding can be contextualized from a practical implication, your studies, you're finding, may not be statistically significant, but it still might have practical implication.

00;56;53;17 - 00;57;16;03
Monsurul Hoq
And this is this study probably, I have to say a great example. It may not be sound on a many, many statistical concept, but it still had a lot of practical significance, whether it's positive or negative, that's that's to the expert. But yes, that's all in there.

00;57;16;06 - 00;57;42;14
Jasmine
I love that ending on what I think is a really key message, especially in psychology. There is a real push and a focus on p values and .05. So I feel so fortunate to have had this conversation with you months. Thanks so much for coming and having this chat. And sharing all this wealth of knowledge that you have in such a practical, kind and accessible way.

00;57;42;16 - 00;57;44;26
Jasmine
Really appreciate your time.

00;57;44;29 - 00;58;11;17
Monsurul Hoq
Thank you. Thank you, Jasmine, for inviting me. And, you know, talking about this paper and p value, something that yes, I'm really interested, as I said at the very beginning, applying the statistics to write the statistics and methods is something my passion and and working with, John Carlin from Melbourne University, that's probably one of the things that I have learned, that the p p value and then not, how we are using it.

00;58;11;17 - 00;58;38;02
Monsurul Hoq
So probably that's something that, you know, whenever I work, those are the probably main thing for me. Like what is your research question is the analysis is that you are doing appropriate for your research question. And can you analyze can you interpret your data based on the context and the practical implication, not only based on the statistics.

00;58;38;09 - 00;58;41;17
Monsurul Hoq
It's coming from a statistician.

00;58;41;19 - 00;58;44;18
Jasmine
That's amazing. Thanks so much.

00;58;44;20 - 00;58;45;01
Monsurul Hoq
Help.